Question 1

  1. The lasso, relative to least squares: iii is true. Lasso is less flexible and hence will give improved prediction accuracy when its increase in bias is less than its decrease in variance.

  2. Ridge regression relative to least squares: Again, iii is true. Ridge regression is less flexible and hence will give improved prediction accuracy when its increase in bias is less than its decrease in variance.

  3. Non-linear methods relative to least squares: ii is true. Non- linear methods are more flexible and hence will give improved prediction accuracy when its increase in variance is less than its decrease in bias.

a-c)

#Part a
set.seed(42)
X = rnorm(n=100)
e = rnorm(n=100)

# B0=1 B1=3 B2=2 B3=1
B = c(1,3,2,1)

#Part B
Y = rep(B[1], 100)
Y = Y + (B[2]*X)+ (B[3]*X^2)+ (B[4]*X^3)+ e

#Part C
x_data = data.frame(X, X^2, X^3, X^4, X^5, X^6, X^7, X^8, X^9, X^10, Y)
colnames(x_data) =c('X', 'X^2', 'X^3', 'X^4', 'X^5', 'X^6', 'X^7', 'X^8', 'X^9', 'X^10', 'Y')
x_data
##               X         X^2           X^3          X^4           X^5
## 1    1.37095845 1.879527064  2.576754e+00 3.532622e+00  4.843078e+00
## 2   -0.56469817 0.318884025 -1.800732e-01 1.016870e-01 -5.742247e-02
## 3    0.36312841 0.131862243  4.788293e-02 1.738765e-02  6.313950e-03
## 4    0.63286260 0.400515077  2.534710e-01 1.604123e-01  1.015190e-01
## 5    0.40426832 0.163432877  6.607074e-02 2.671031e-02  1.079813e-02
## 6   -0.10612452 0.011262413 -1.195218e-03 1.268419e-04 -1.346104e-05
## 7    1.51152200 2.284698749  3.453372e+00 5.219848e+00  7.889916e+00
## 8   -0.09465904 0.008960334 -8.481766e-04 8.028758e-05 -7.599945e-06
## 9    2.01842371 4.074034289  8.223127e+00 1.659776e+01  3.350130e+01
## 10  -0.06271410 0.003933058 -2.466582e-04 1.546895e-05 -9.701211e-07
## 11   1.30486965 1.702684815  2.221782e+00 2.899136e+00  3.782994e+00
## 12   2.28664539 5.228747152  1.195629e+01 2.733980e+01  6.251642e+01
## 13  -1.38886070 1.928934047 -2.679021e+00 3.720787e+00 -5.167654e+00
## 14  -0.27878877 0.077723177 -2.166835e-02 6.040892e-03 -1.684133e-03
## 15  -0.13332134 0.017774579 -2.369731e-03 3.159356e-04 -4.212096e-05
## 16   0.63595040 0.404432909  2.571993e-01 1.635660e-01  1.040198e-01
## 17  -0.28425292 0.080799723 -2.296756e-02 6.528595e-03 -1.855772e-03
## 18  -2.65645542 7.056755403 -1.874596e+01 4.979780e+01 -1.322856e+02
## 19  -2.44046693 5.955878829 -1.453513e+01 3.547249e+01 -8.656945e+01
## 20   1.32011335 1.742699246  2.300561e+00 3.037001e+00  4.009185e+00
## 21  -0.30663859 0.094027227 -2.883238e-02 8.841119e-03 -2.711028e-03
## 22  -1.78130843 3.173059737 -5.652198e+00 1.006831e+01 -1.793476e+01
## 23  -0.17191736 0.029555577 -5.081117e-03 8.735321e-04 -1.501753e-04
## 24   1.21467470 1.475434625  1.792173e+00 2.176907e+00  2.644234e+00
## 25   1.89519346 3.591758256  6.807077e+00 1.290073e+01  2.444937e+01
## 26  -0.43046913 0.185303673 -7.976751e-02 3.433745e-02 -1.478121e-02
## 27  -0.25726938 0.066187535 -1.702803e-02 4.380790e-03 -1.127043e-03
## 28  -1.76316309 3.108744065 -5.481223e+00 9.664290e+00 -1.703972e+01
## 29   0.46009735 0.211689576  9.739781e-02 4.481248e-02  2.061810e-02
## 30  -0.63999488 0.409593441 -2.621377e-01 1.677668e-01 -1.073699e-01
## 31   0.45545012 0.207434815  9.447621e-02 4.302920e-02  1.959766e-02
## 32   0.70483734 0.496795672  3.501601e-01 2.468059e-01  1.739580e-01
## 33   1.03510352 1.071439301  1.109051e+00 1.147982e+00  1.188280e+00
## 34  -0.60892638 0.370791331 -2.257846e-01 1.374862e-01 -8.371898e-02
## 35   0.50495512 0.254979677  1.287533e-01 6.501464e-02  3.282947e-02
## 36  -1.71700868 2.948118804 -5.061946e+00 8.691404e+00 -1.492322e+01
## 37  -0.78445901 0.615375936 -4.827372e-01 3.786875e-01 -2.970649e-01
## 38  -0.85090759 0.724043734 -6.160943e-01 5.242393e-01 -4.460792e-01
## 39  -2.41420765 5.828398577 -1.407096e+01 3.397023e+01 -8.201119e+01
## 40   0.03612261 0.001304843  4.713432e-05 1.702615e-06  6.150288e-08
## 41   0.20599860 0.042435423  8.741638e-03 1.800765e-03  3.709551e-04
## 42  -0.36105730 0.130362373 -4.706829e-02 1.699435e-02 -6.135933e-03
## 43   0.75816324 0.574811492  4.358009e-01 3.304083e-01  2.505034e-01
## 44  -0.72670483 0.528099906 -3.837728e-01 2.788895e-01 -2.026704e-01
## 45  -1.36828104 1.872193017 -2.561686e+00 3.505107e+00 -4.795971e+00
## 46   0.43281803 0.187331444  8.108043e-02 3.509307e-02  1.518891e-02
## 47  -0.81139318 0.658358886 -5.341879e-01 4.334364e-01 -3.516874e-01
## 48   1.44410126 2.085428454  3.011570e+00 4.349012e+00  6.280413e+00
## 49  -0.43144620 0.186145826 -8.031191e-02 3.465027e-02 -1.494973e-02
## 50   0.65564788 0.429874147  2.818461e-01 1.847918e-01  1.211583e-01
## 51   0.32192527 0.103635876  3.336301e-02 1.074039e-02  3.457604e-03
## 52  -0.78383894 0.614403485 -4.815934e-01 3.774916e-01 -2.958926e-01
## 53   1.57572752 2.482917217  3.912401e+00 6.164878e+00  9.714168e+00
## 54   0.64289931 0.413319517  2.657228e-01 1.708330e-01  1.098284e-01
## 55   0.08976065 0.008056974  7.231992e-04 6.491482e-05  5.826797e-06
## 56   0.27655075 0.076480316  2.115069e-02 5.849239e-03  1.617611e-03
## 57   0.67928882 0.461433296  3.134465e-01 2.129207e-01  1.446346e-01
## 58   0.08983289 0.008069948  7.249467e-04 6.512405e-05  5.850282e-06
## 59  -2.99309008 8.958588246 -2.681386e+01 8.025630e+01 -2.402143e+02
## 60   0.28488295 0.081158297  2.312062e-02 6.586669e-03  1.876430e-03
## 61  -0.36723464 0.134861283 -4.952574e-02 1.818757e-02 -6.679104e-03
## 62   0.18523056 0.034310362  6.355328e-03 1.177201e-03  2.180536e-04
## 63   0.58182373 0.338518850  1.969583e-01 1.145950e-01  6.667410e-02
## 64   1.39973683 1.959263186  2.742453e+00 3.838712e+00  5.373187e+00
## 65  -0.72729206 0.528953740 -3.847039e-01 2.797921e-01 -2.034905e-01
## 66   1.30254263 1.696617308  2.209916e+00 2.878510e+00  3.749382e+00
## 67   0.33584812 0.112793960  3.788164e-02 1.272248e-02  4.272820e-03
## 68   1.03850610 1.078494917  1.120024e+00 1.163151e+00  1.207940e+00
## 69   0.92072857 0.847741096  7.805394e-01 7.186650e-01  6.616954e-01
## 70   0.72087816 0.519665326  3.746154e-01 2.700521e-01  1.946746e-01
## 71  -1.04311894 1.088097120 -1.135015e+00 1.183955e+00 -1.235006e+00
## 72  -0.09018639 0.008133584 -7.335386e-04 6.615519e-05 -5.966298e-06
## 73   0.62351816 0.388774898  2.424082e-01 1.511459e-01  9.424223e-02
## 74  -0.95352336 0.909206794 -8.669499e-01 8.266570e-01 -7.882368e-01
## 75  -0.54282881 0.294663122 -1.599516e-01 8.682636e-02 -4.713185e-02
## 76   0.58099650 0.337556930  1.961194e-01 1.139447e-01  6.620146e-02
## 77   0.76817874 0.590098573  4.533012e-01 3.482163e-01  2.674924e-01
## 78   0.46376759 0.215080376  9.974731e-02 4.625957e-02  2.145369e-02
## 79  -0.88577630 0.784599649 -6.949798e-01 6.155966e-01 -5.452809e-01
## 80  -1.09978090 1.209518025 -1.330205e+00 1.462934e+00 -1.608907e+00
## 81   1.51270701 2.288282498  3.461501e+00 5.236237e+00  7.920892e+00
## 82   0.25792144 0.066523468  1.715783e-02 4.425372e-03  1.141398e-03
## 83   0.08844023 0.007821674  6.917507e-04 6.117859e-05  5.410648e-06
## 84  -0.12089654 0.014615973 -1.767021e-03 2.136267e-04 -2.582672e-05
## 85  -1.19432890 1.426421510 -1.703616e+00 2.034678e+00 -2.430075e+00
## 86   0.61199690 0.374540203  2.292174e-01 1.402804e-01  8.585115e-02
## 87  -0.21713985 0.047149713 -1.023808e-02 2.223095e-03 -4.827226e-04
## 88  -0.18275671 0.033400014 -6.104076e-03 1.115561e-03 -2.038762e-04
## 89   0.93334633 0.871135369  8.130710e-01 7.588768e-01  7.082949e-01
## 90   0.82177311 0.675311045  5.549525e-01 4.560450e-01  3.747655e-01
## 91   1.39211638 1.937988004  2.697905e+00 3.755798e+00  5.228507e+00
## 92  -0.47617392 0.226741605 -1.079684e-01 5.141176e-02 -2.448094e-02
## 93   0.65034856 0.422953250  2.750670e-01 1.788895e-01  1.163405e-01
## 94   1.39111046 1.935188302  2.692061e+00 3.744954e+00  5.209644e+00
## 95  -1.11078888 1.233851935 -1.370549e+00 1.522391e+00 -1.691055e+00
## 96  -0.86079259 0.740963878 -6.378162e-01 5.490275e-01 -4.725988e-01
## 97  -1.13173868 1.280832442 -1.449568e+00 1.640532e+00 -1.856653e+00
## 98  -1.45921400 2.129305496 -3.107112e+00 4.533942e+00 -6.615991e+00
## 99   0.07998255 0.006397209  5.116651e-04 4.092428e-05  3.273228e-06
## 100  0.65320434 0.426675909  2.787066e-01 1.820523e-01  1.189174e-01
##              X^6           X^7          X^8           X^9         X^10
## 1   6.639659e+00  9.102696e+00 1.247942e+01  1.710876e+01 2.345540e+01
## 2   3.242637e-02 -1.831111e-02 1.034025e-02 -5.839120e-03 3.297341e-03
## 3   2.292775e-03  8.325716e-04 3.023304e-04  1.097848e-04 3.986597e-05
## 4   6.424756e-02  4.065988e-02 2.573211e-02  1.628489e-02 1.030610e-02
## 5   4.365342e-03  1.764770e-03 7.134404e-04  2.884214e-04 1.165996e-04
## 6   1.428546e-06 -1.516038e-07 1.608888e-08 -1.707424e-09 1.811996e-10
## 7   1.192578e+01  1.802608e+01 2.724682e+01  4.118416e+01 6.225077e+01
## 8   7.194035e-07 -6.809804e-08 6.446095e-09 -6.101812e-10 5.775916e-11
## 9   6.761982e+01  1.364855e+02 2.754855e+02  5.560464e+02 1.122337e+03
## 10  6.084027e-08 -3.815543e-09 2.392883e-10 -1.500675e-11 9.411349e-13
## 11  4.936314e+00  6.441247e+00 8.404987e+00  1.096741e+01 1.431104e+01
## 12  1.429529e+02  3.268826e+02 7.474645e+02  1.709186e+03 3.908303e+03
## 13  7.177152e+00 -9.968064e+00 1.384425e+01 -1.922774e+01 2.670465e+01
## 14  4.695173e-04 -1.308962e-04 3.649238e-05 -1.017367e-05 2.836304e-06
## 15  5.615623e-06 -7.486824e-07 9.981533e-08 -1.330751e-08 1.774176e-09
## 16  6.615146e-02  4.206905e-02 2.675383e-02  1.701411e-02 1.082013e-02
## 17  5.275087e-04 -1.499459e-04 4.262256e-05 -1.211559e-05 3.443891e-06
## 18  3.514109e+02 -9.335073e+02 2.479821e+03 -6.587533e+03 1.749949e+04
## 19  2.112699e+02 -5.155971e+02 1.258298e+03 -3.070834e+03 7.494269e+03
## 20  5.292579e+00  6.986804e+00 9.223373e+00  1.217590e+01 1.607357e+01
## 21  8.313060e-04 -2.549105e-04 7.816539e-05 -2.396853e-05 7.349675e-06
## 22  3.194734e+01 -5.690807e+01 1.013708e+02 -1.805727e+02 3.216557e+02
## 23  2.581775e-05 -4.438519e-06 7.630584e-07 -1.311830e-07 2.255263e-08
## 24  3.211884e+00  3.901395e+00 4.738926e+00  5.756253e+00 6.991975e+00
## 25  4.633629e+01  8.781624e+01 1.664288e+02  3.154147e+02 5.977719e+02
## 26  6.362856e-03 -2.739013e-03 1.179061e-03 -5.075492e-04 2.184843e-04
## 27  2.899537e-04 -7.459620e-05 1.919132e-05 -4.937339e-06 1.270226e-06
## 28  3.004380e+01 -5.297212e+01 9.339849e+01 -1.646768e+02 2.903520e+02
## 29  9.486334e-03  4.364637e-03 2.008158e-03  9.239482e-04 4.251061e-04
## 30  6.871618e-02 -4.397800e-02 2.814569e-02 -1.801310e-02 1.152829e-02
## 31  8.925755e-03  4.065236e-03 1.851512e-03  8.432715e-04 3.840681e-04
## 32  1.226121e-01  8.642160e-02 6.091317e-02  4.293388e-02 3.026140e-02
## 33  1.229993e+00  1.273170e+00 1.317863e+00  1.364125e+00 1.412010e+00
## 34  5.097870e-02 -3.104227e-02 1.890246e-02 -1.151021e-02 7.008868e-03
## 35  1.657741e-02  8.370848e-03 4.226903e-03  2.134396e-03 1.077774e-03
## 36  2.562329e+01 -4.399542e+01 7.554051e+01 -1.297037e+02 2.227024e+02
## 37  2.330352e-01 -1.828066e-01 1.434043e-01 -1.124948e-01 8.824753e-02
## 38  3.795722e-01 -3.229809e-01 2.748269e-01 -2.338523e-01 1.989867e-01
## 39  1.979920e+02 -4.779939e+02 1.153977e+03 -2.785939e+03 6.725835e+03
## 40  2.221644e-09  8.025158e-11 2.898896e-12  1.047157e-13 3.782604e-15
## 41  7.641623e-05  1.574164e-05 3.242755e-06  6.680030e-07 1.376077e-07
## 42  2.215424e-03 -7.998948e-04 2.888079e-04 -1.042762e-04 3.764968e-05
## 43  1.899225e-01  1.439922e-01 1.091696e-01  8.276839e-02 6.275195e-02
## 44  1.472815e-01 -1.070302e-01 7.777936e-02 -5.652264e-02 4.107527e-02
## 45  6.562236e+00 -8.978983e+00 1.228577e+01 -1.681039e+01 2.300134e+01
## 46  6.574035e-03  2.845361e-03 1.231524e-03  5.330256e-04 2.307031e-04
## 47  2.853567e-01 -2.315365e-01 1.878671e-01 -1.524341e-01 1.236840e-01
## 48  9.069553e+00  1.309735e+01 1.891390e+01  2.731359e+01 3.944359e+01
## 49  6.450003e-03 -2.782829e-03 1.200641e-03 -5.180120e-04 2.234943e-04
## 50  7.943721e-02  5.208284e-02 3.414800e-02  2.238907e-02 1.467934e-02
## 51  1.113090e-03  3.583319e-04 1.153561e-04  3.713604e-05 1.195503e-05
## 52  2.319322e-01 -1.817975e-01 1.424999e-01 -1.116970e-01 8.755246e-02
## 53  1.530688e+01  2.411947e+01 3.800572e+01  5.988666e+01 9.436506e+01
## 54  7.060862e-02  4.539423e-02 2.918392e-02  1.876232e-02 1.206228e-02
## 55  5.230170e-07  4.694635e-08 4.213934e-09  3.782455e-10 3.395156e-11
## 56  4.473516e-04  1.237154e-04 3.421359e-05  9.461795e-06 2.616666e-06
## 57  9.824869e-02  6.673924e-02 4.533522e-02  3.079571e-02 2.091918e-02
## 58  5.255477e-07  4.721147e-08 4.241142e-09  3.809941e-10 3.422580e-11
## 59  7.189832e+02 -2.151981e+03 6.441074e+03 -1.927872e+04 5.770293e+04
## 60  5.345629e-04  1.522878e-04 4.338421e-05  1.235942e-05 3.520989e-06
## 61  2.452798e-03 -9.007526e-04 3.307875e-04 -1.214766e-04 4.461043e-05
## 62  4.039019e-05  7.481498e-06 1.385802e-06  2.566929e-07 4.754737e-08
## 63  3.879257e-02  2.257044e-02 1.313202e-02  7.640519e-03 4.445435e-03
## 64  7.521048e+00  1.052749e+01 1.473571e+01  2.062612e+01 2.887114e+01
## 65  1.479971e-01 -1.076371e-01 7.828360e-02 -5.693504e-02 4.140840e-02
## 66  4.883730e+00  6.361267e+00 8.285821e+00  1.079264e+01 1.405787e+01
## 67  1.435019e-03  4.819483e-04 1.618614e-04  5.436086e-05 1.825699e-05
## 68  1.254453e+00  1.302757e+00 1.352921e+00  1.405017e+00 1.459118e+00
## 69  6.092418e-01  5.609464e-01 5.164793e-01  4.755373e-01 4.378408e-01
## 70  1.403367e-01  1.011657e-01 7.292811e-02  5.257228e-02 3.789821e-02
## 71  1.288258e+00 -1.343807e+00 1.401750e+00 -1.462192e+00 1.525240e+00
## 72  5.380788e-07 -4.852739e-08 4.376510e-09 -3.947016e-10 3.559671e-11
## 73  5.876174e-02  3.663901e-02 2.284509e-02  1.424433e-02 8.881597e-03
## 74  7.516022e-01 -7.166702e-01 6.833618e-01 -6.516014e-01 6.213172e-01
## 75  2.558452e-02 -1.388802e-02 7.538816e-03 -4.092287e-03 2.221411e-03
## 76  3.846282e-02  2.234676e-02 1.298339e-02  7.543304e-03 4.382633e-03
## 77  2.054820e-01  1.578469e-01 1.212546e-01  9.314521e-02 7.155217e-02
## 78  9.949525e-03  4.614267e-03 2.139948e-03  9.924384e-04 4.602607e-04
## 79  4.829969e-01 -4.278272e-01 3.789592e-01 -3.356731e-01 2.973312e-01
## 80  1.769445e+00 -1.946002e+00 2.140175e+00 -2.353724e+00 2.588581e+00
## 81  1.198199e+01  1.812524e+01 2.741818e+01  4.147567e+01 6.274053e+01
## 82  2.943911e-04  7.592977e-05 1.958392e-05  5.051112e-06 1.302790e-06
## 83  4.785190e-07  4.232033e-08 3.742819e-09  3.310158e-10 2.927511e-11
## 84  3.122361e-06 -3.774827e-07 4.563635e-08 -5.517277e-09 6.670197e-10
## 85  2.902309e+00 -3.466311e+00 4.139916e+00 -4.944421e+00 5.905265e+00
## 86  5.254064e-02  3.215471e-02 1.967858e-02  1.204323e-02 7.370420e-03
## 87  1.048183e-04 -2.276023e-05 4.942153e-06 -1.073138e-06 2.330211e-07
## 88  3.725975e-05 -6.809469e-06 1.244476e-06 -2.274364e-07 4.156552e-08
## 89  6.610844e-01  6.170207e-01 5.758940e-01  5.375086e-01 5.016817e-01
## 90  3.079722e-01  2.530833e-01 2.079770e-01  1.709099e-01 1.404492e-01
## 91  7.278691e+00  1.013278e+01 1.410601e+01  1.963721e+01 2.733729e+01
## 92  1.165718e-02 -5.550847e-03 2.643169e-03 -1.258608e-03 5.993163e-04
## 93  7.566188e-02  4.920659e-02 3.200144e-02  2.081209e-02 1.353511e-02
## 94  7.247191e+00  1.008164e+01 1.402468e+01  1.950988e+01 2.714039e+01
## 95  1.878405e+00 -2.086511e+00 2.317673e+00 -2.574446e+00 2.859665e+00
## 96  4.068095e-01 -3.501786e-01 3.014312e-01 -2.594697e-01 2.233496e-01
## 97  2.101246e+00 -2.378062e+00 2.691344e+00 -3.045899e+00 3.447161e+00
## 98  9.654147e+00 -1.408747e+01 2.055663e+01 -2.999652e+01 4.377134e+01
## 99  2.618012e-07  2.093953e-08 1.674797e-09  1.339545e-10 1.071402e-11
## 100 7.767734e-02  5.073918e-02 3.314305e-02  2.164919e-02 1.414134e-02
##                Y
## 1    12.64964835
## 2     0.80835140
## 3     1.39778400
## 4     5.80157088
## 5     1.93896805
## 6     0.80846987
## 7    13.13508002
## 8     0.61074520
## 9    23.61466017
## 10    0.93863812
## 11   10.51666779
## 12   30.38179379
## 13   -2.47316994
## 14   -0.20680543
## 15   -1.02788366
## 16    3.59158255
## 17   -0.22677713
## 18   -8.89992060
## 19  -10.30688467
## 20   10.88355528
## 21   -1.25431877
## 22   -5.12043964
## 23    0.66298036
## 24    8.39042732
## 25   20.67435104
## 26   -0.42882644
## 27   -0.27013271
## 28   -5.57790175
## 29    1.67632108
## 30   -0.18341901
## 31    3.44331681
## 32    3.96538614
## 33    7.35730265
## 34    0.81190856
## 35    4.59343376
## 36   -4.41384777
## 37   -0.72268191
## 38    0.48076877
## 39   -9.12651981
## 40    1.05855516
## 41    1.62550099
## 42   -0.75719445
## 43    4.41522963
## 44   -0.53713230
## 45   -2.33601217
## 46    3.86758341
## 47   -1.13264251
## 48   12.08156152
## 49    0.69450371
## 50    3.05216961
## 51    2.16571208
## 52   -2.15584805
## 53   15.77258753
## 54    3.74741408
## 55    0.81827376
## 56    0.76551123
## 57    4.26641748
## 58    0.48608132
## 59  -17.40944773
## 60    3.32776132
## 61   -0.05703297
## 62    0.55888536
## 63    3.78267406
## 64   11.49745127
## 65    0.08134099
## 66   11.94320081
## 67    1.27832141
## 68    7.84718198
## 69    6.32310540
## 70    5.47214611
## 71   -1.31795543
## 72    1.58159354
## 73    2.14545663
## 74    0.78035252
## 75    0.66566615
## 76    3.46344676
## 77    3.48902741
## 78    3.56421953
## 79   -0.29991550
## 80   -1.21686709
## 81   13.72764289
## 82    1.33986011
## 83    1.65046252
## 84    0.95942965
## 85   -1.71301947
## 86    2.47805189
## 87    1.13339063
## 88    1.06662245
## 89    5.51907413
## 90    3.77630572
## 91   11.95518855
## 92   -0.42809498
## 93    4.32463092
## 94   10.44176619
## 95   -2.19438222
## 96    0.34750864
## 97   -0.87934387
## 98   -1.63965586
## 99    3.06848219
## 100   4.22049282
a<-regsubsets(Y~., data = x_data, nvmax = 10)

mysummary(a)
## Subset selection object
## Call: regsubsets.formula(Y ~ ., data = x_data, nvmax = 10)
## 10 Variables  (and intercept)
##        Forced in Forced out
## X          FALSE      FALSE
## `X^2`      FALSE      FALSE
## `X^3`      FALSE      FALSE
## `X^4`      FALSE      FALSE
## `X^5`      FALSE      FALSE
## `X^6`      FALSE      FALSE
## `X^7`      FALSE      FALSE
## `X^8`      FALSE      FALSE
## `X^9`      FALSE      FALSE
## `X^10`     FALSE      FALSE
## 1 subsets of each size up to 10
## Selection Algorithm: exhaustive
##           X   `X^2` `X^3` `X^4` `X^5` `X^6` `X^7` `X^8` `X^9` `X^10`
## 1  ( 1 )  "*" " "   " "   " "   " "   " "   " "   " "   " "   " "   
## 2  ( 1 )  " " " "   "*"   "*"   " "   " "   " "   " "   " "   " "   
## 3  ( 1 )  "*" "*"   "*"   " "   " "   " "   " "   " "   " "   " "   
## 4  ( 1 )  "*" "*"   "*"   " "   "*"   " "   " "   " "   " "   " "   
## 5  ( 1 )  "*" "*"   " "   " "   "*"   " "   "*"   " "   "*"   " "   
## 6  ( 1 )  "*" "*"   " "   "*"   "*"   " "   "*"   " "   "*"   " "   
## 7  ( 1 )  "*" "*"   " "   " "   "*"   "*"   "*"   "*"   "*"   " "   
## 8  ( 1 )  "*" "*"   " "   "*"   "*"   "*"   "*"   "*"   "*"   " "   
## 9  ( 1 )  "*" "*"   " "   "*"   "*"   "*"   "*"   "*"   "*"   "*"   
## 10  ( 1 ) "*" "*"   "*"   "*"   "*"   "*"   "*"   "*"   "*"   "*"   
## [1] "Cps: "
##  [1] 1043.992234  433.630846   21.318293   19.561749    5.568909
##  [6]    6.763693    5.528537    7.337124    9.080219   11.000000

## [1] "Optimal model variable count:"
## [1] 7

## [1] "BICs: "
##  [1] -153.7239 -226.1567 -375.3712 -374.1372 -385.3111 -381.5702 -380.5148
##  [8] -376.1237 -371.8065 -367.2914

## [1] "Optimal model variable count:"
## [1] 5
## [1] "R-squared: "
##  [1] 0.8039415 0.9092568 0.9805113 0.9811573 0.9839078 0.9840463 0.9846027
##  [8] 0.9846356 0.9846798 0.9846936

## [1] "Optimal model variable count:"
## [1] 10

The 7th model is best according to Cp. The 5th model is best according to BIC. The 8th model is best according to R-Squared.

d)

a2<-regsubsets(Y~., data = x_data, method = "forward", nvmax = 10)
mysummary(a2)
## Subset selection object
## Call: regsubsets.formula(Y ~ ., data = x_data, method = "forward", 
##     nvmax = 10)
## 10 Variables  (and intercept)
##        Forced in Forced out
## X          FALSE      FALSE
## `X^2`      FALSE      FALSE
## `X^3`      FALSE      FALSE
## `X^4`      FALSE      FALSE
## `X^5`      FALSE      FALSE
## `X^6`      FALSE      FALSE
## `X^7`      FALSE      FALSE
## `X^8`      FALSE      FALSE
## `X^9`      FALSE      FALSE
## `X^10`     FALSE      FALSE
## 1 subsets of each size up to 10
## Selection Algorithm: forward
##           X   `X^2` `X^3` `X^4` `X^5` `X^6` `X^7` `X^8` `X^9` `X^10`
## 1  ( 1 )  "*" " "   " "   " "   " "   " "   " "   " "   " "   " "   
## 2  ( 1 )  "*" "*"   " "   " "   " "   " "   " "   " "   " "   " "   
## 3  ( 1 )  "*" "*"   "*"   " "   " "   " "   " "   " "   " "   " "   
## 4  ( 1 )  "*" "*"   "*"   " "   "*"   " "   " "   " "   " "   " "   
## 5  ( 1 )  "*" "*"   "*"   " "   "*"   " "   " "   " "   "*"   " "   
## 6  ( 1 )  "*" "*"   "*"   " "   "*"   " "   "*"   " "   "*"   " "   
## 7  ( 1 )  "*" "*"   "*"   "*"   "*"   " "   "*"   " "   "*"   " "   
## 8  ( 1 )  "*" "*"   "*"   "*"   "*"   " "   "*"   " "   "*"   "*"   
## 9  ( 1 )  "*" "*"   "*"   "*"   "*"   "*"   "*"   " "   "*"   "*"   
## 10  ( 1 ) "*" "*"   "*"   "*"   "*"   "*"   "*"   "*"   "*"   "*"   
## [1] "Cps: "
##  [1] 1043.992234  710.611218   21.318293   19.561749   13.822436
##  [6]    7.469641    8.763127    7.984529    9.697202   11.000000

## [1] "Optimal model variable count:"
## [1] 6

## [1] "BICs: "
##  [1] -153.7239 -183.9605 -375.3712 -374.1372 -376.8578 -380.8120 -376.9656
##  [8] -375.4016 -371.1162 -367.2914

## [1] "Optimal model variable count:"
## [1] 6
## [1] "R-squared: "
##  [1] 0.8039415 0.8616211 0.9805113 0.9811573 0.9824883 0.9839249 0.9840464
##  [8] 0.9845243 0.9845737 0.9846936

## [1] "Optimal model variable count:"
## [1] 10

The 6th model is best according to Cp. The 6th model is best according to BIC. The 8th model is best according to R-Squared.

a3<-regsubsets(Y~., data = x_data, method = "backward", nvmax = 10)
mysummary(a3)
## Subset selection object
## Call: regsubsets.formula(Y ~ ., data = x_data, method = "backward", 
##     nvmax = 10)
## 10 Variables  (and intercept)
##        Forced in Forced out
## X          FALSE      FALSE
## `X^2`      FALSE      FALSE
## `X^3`      FALSE      FALSE
## `X^4`      FALSE      FALSE
## `X^5`      FALSE      FALSE
## `X^6`      FALSE      FALSE
## `X^7`      FALSE      FALSE
## `X^8`      FALSE      FALSE
## `X^9`      FALSE      FALSE
## `X^10`     FALSE      FALSE
## 1 subsets of each size up to 10
## Selection Algorithm: backward
##           X   `X^2` `X^3` `X^4` `X^5` `X^6` `X^7` `X^8` `X^9` `X^10`
## 1  ( 1 )  "*" " "   " "   " "   " "   " "   " "   " "   " "   " "   
## 2  ( 1 )  "*" "*"   " "   " "   " "   " "   " "   " "   " "   " "   
## 3  ( 1 )  "*" "*"   " "   " "   "*"   " "   " "   " "   " "   " "   
## 4  ( 1 )  "*" "*"   " "   " "   "*"   " "   "*"   " "   " "   " "   
## 5  ( 1 )  "*" "*"   " "   " "   "*"   " "   "*"   " "   "*"   " "   
## 6  ( 1 )  "*" "*"   " "   " "   "*"   "*"   "*"   " "   "*"   " "   
## 7  ( 1 )  "*" "*"   " "   " "   "*"   "*"   "*"   "*"   "*"   " "   
## 8  ( 1 )  "*" "*"   " "   "*"   "*"   "*"   "*"   "*"   "*"   " "   
## 9  ( 1 )  "*" "*"   " "   "*"   "*"   "*"   "*"   "*"   "*"   "*"   
## 10  ( 1 ) "*" "*"   "*"   "*"   "*"   "*"   "*"   "*"   "*"   "*"   
## [1] "Cps: "
##  [1] 1043.992234  710.611218  122.597547   56.309625    5.568909
##  [6]    7.181454    5.528537    7.337124    9.080219   11.000000

## [1] "Optimal model variable count:"
## [1] 7

## [1] "BICs: "
##  [1] -153.7239 -183.9605 -311.5148 -345.2135 -385.3111 -381.1208 -380.5148
##  [8] -376.1237 -371.8065 -367.2914

## [1] "Optimal model variable count:"
## [1] 5
## [1] "R-squared: "
##  [1] 0.8039415 0.8616211 0.9630930 0.9748373 0.9839078 0.9839744 0.9846027
##  [8] 0.9846356 0.9846798 0.9846936

## [1] "Optimal model variable count:"
## [1] 10

The 7th model is best according to Cp. The 5th model is best according to BIC. The 8th model is best according to R-Squared.

The model with the most amount of variables always has the larges R-squared value becasue, unlike Cp and BIC, there is no punishment for adding cofactors in the calculation for R-squared. That being said, it is not suprising that the best model according to the R-Squared value is consistently the model with 8 variables. That model is the same between the exaustive search and backward search, which is the model that includes every X except X^3 and X^10. For forward stepwise, that model is the model that includes every X except X^6 and X^8.

According to Cp, the best model found in the exaustive search was the model with 7 variables, which included every X except X^3, X^4, and X^10. For the forward stepwise it was the model includeing 6 variables (X to the 1,2,3,5,7, and 9th power). For the backward stepwise it was the same 7 variable model found from the exaustive search.

According to BIC, the best model found in the exaustive search was the model with 5 variables (X to the 1,2,5,7, and 9th power). The same 5 variable model was found in when backward stepwise was applied. For the forward stepwise, however, the 6 variable model with X to the 1,2,3,5,7, and 9th powers was the best.

e)

#fit lasso reg. model
x_dataM = as.matrix(x_data)
grid = seq(0,10, length =100)
lasso.mod=glmnet(x_dataM[,-11],x_dataM[ ,11],alpha=1, lambda = grid)
plot(lasso.mod, label=TRUE)

#Cross-validation
set.seed(42)
#x_dataf = as.data.frame(x_data) 
cv.out=cv.glmnet(x_dataM[,-11],x_dataM[ ,11],alpha=1)
plot(cv.out)

#find lambda corresponding to smallest mse
bestlam =cv.out$lambda.min
bestlam
## [1] 0.168265
#extract coeficients for model with this lambda
coef(cv.out, s = "lambda.min")
## 11 x 1 sparse Matrix of class "dgCMatrix"
##                     1
## (Intercept) 1.1188429
## X           3.0702398
## X^2         1.7706815
## X^3         0.9109737
## X^4         .        
## X^5         .        
## X^6         .        
## X^7         .        
## X^8         .        
## X^9         .        
## X^10        .

The lambda that gives the smallest cross-validated mean-squared error is 0.168265. This lambda corresponds with the point on the cross-validation graph whose log(Lambda) = log(0.168265) = -0.77401, which is the point between the two calculated vertical dotted lines. The coefficient values can be found in the 11 x 1 spase matrix above. Notice that X, X^2, and X^3 are the only covariates with non-zero coefficients, and their coefficients are 3.07, 1.77, and 0.91, correspondingly. Remember our data was created using the real coefficient valeus of 3, 2, and 1. Thus, this model is fairly accurate in its assumptions. It overestimates the X coefficient, and underestimate both the X^2 and the X^3 coefficient. The fact that these particular covariates have non-zero coeficients for the optimal model is not suprising, since our data was generated using exclusively these three variables.

Data Generation)

# B0=1 B7=2.5
B_2 = c(1,2.5)

Y_2 = rep(B_2[1], 100)
Y_2 = Y_2 + (B_2[2]*X^7) + e


x_data_2 = data.frame(X, X^2, X^3, X^4, X^5, X^6, X^7, X^8, X^9, X^10, Y_2)
colnames(x_data) =c('X', 'X^2', 'X^3', 'X^4', 'X^5', 'X^6', 'X^7', 'X^8', 'X^9', 'X^10', 'Y_2')
x_data
##               X         X^2           X^3          X^4           X^5
## 1    1.37095845 1.879527064  2.576754e+00 3.532622e+00  4.843078e+00
## 2   -0.56469817 0.318884025 -1.800732e-01 1.016870e-01 -5.742247e-02
## 3    0.36312841 0.131862243  4.788293e-02 1.738765e-02  6.313950e-03
## 4    0.63286260 0.400515077  2.534710e-01 1.604123e-01  1.015190e-01
## 5    0.40426832 0.163432877  6.607074e-02 2.671031e-02  1.079813e-02
## 6   -0.10612452 0.011262413 -1.195218e-03 1.268419e-04 -1.346104e-05
## 7    1.51152200 2.284698749  3.453372e+00 5.219848e+00  7.889916e+00
## 8   -0.09465904 0.008960334 -8.481766e-04 8.028758e-05 -7.599945e-06
## 9    2.01842371 4.074034289  8.223127e+00 1.659776e+01  3.350130e+01
## 10  -0.06271410 0.003933058 -2.466582e-04 1.546895e-05 -9.701211e-07
## 11   1.30486965 1.702684815  2.221782e+00 2.899136e+00  3.782994e+00
## 12   2.28664539 5.228747152  1.195629e+01 2.733980e+01  6.251642e+01
## 13  -1.38886070 1.928934047 -2.679021e+00 3.720787e+00 -5.167654e+00
## 14  -0.27878877 0.077723177 -2.166835e-02 6.040892e-03 -1.684133e-03
## 15  -0.13332134 0.017774579 -2.369731e-03 3.159356e-04 -4.212096e-05
## 16   0.63595040 0.404432909  2.571993e-01 1.635660e-01  1.040198e-01
## 17  -0.28425292 0.080799723 -2.296756e-02 6.528595e-03 -1.855772e-03
## 18  -2.65645542 7.056755403 -1.874596e+01 4.979780e+01 -1.322856e+02
## 19  -2.44046693 5.955878829 -1.453513e+01 3.547249e+01 -8.656945e+01
## 20   1.32011335 1.742699246  2.300561e+00 3.037001e+00  4.009185e+00
## 21  -0.30663859 0.094027227 -2.883238e-02 8.841119e-03 -2.711028e-03
## 22  -1.78130843 3.173059737 -5.652198e+00 1.006831e+01 -1.793476e+01
## 23  -0.17191736 0.029555577 -5.081117e-03 8.735321e-04 -1.501753e-04
## 24   1.21467470 1.475434625  1.792173e+00 2.176907e+00  2.644234e+00
## 25   1.89519346 3.591758256  6.807077e+00 1.290073e+01  2.444937e+01
## 26  -0.43046913 0.185303673 -7.976751e-02 3.433745e-02 -1.478121e-02
## 27  -0.25726938 0.066187535 -1.702803e-02 4.380790e-03 -1.127043e-03
## 28  -1.76316309 3.108744065 -5.481223e+00 9.664290e+00 -1.703972e+01
## 29   0.46009735 0.211689576  9.739781e-02 4.481248e-02  2.061810e-02
## 30  -0.63999488 0.409593441 -2.621377e-01 1.677668e-01 -1.073699e-01
## 31   0.45545012 0.207434815  9.447621e-02 4.302920e-02  1.959766e-02
## 32   0.70483734 0.496795672  3.501601e-01 2.468059e-01  1.739580e-01
## 33   1.03510352 1.071439301  1.109051e+00 1.147982e+00  1.188280e+00
## 34  -0.60892638 0.370791331 -2.257846e-01 1.374862e-01 -8.371898e-02
## 35   0.50495512 0.254979677  1.287533e-01 6.501464e-02  3.282947e-02
## 36  -1.71700868 2.948118804 -5.061946e+00 8.691404e+00 -1.492322e+01
## 37  -0.78445901 0.615375936 -4.827372e-01 3.786875e-01 -2.970649e-01
## 38  -0.85090759 0.724043734 -6.160943e-01 5.242393e-01 -4.460792e-01
## 39  -2.41420765 5.828398577 -1.407096e+01 3.397023e+01 -8.201119e+01
## 40   0.03612261 0.001304843  4.713432e-05 1.702615e-06  6.150288e-08
## 41   0.20599860 0.042435423  8.741638e-03 1.800765e-03  3.709551e-04
## 42  -0.36105730 0.130362373 -4.706829e-02 1.699435e-02 -6.135933e-03
## 43   0.75816324 0.574811492  4.358009e-01 3.304083e-01  2.505034e-01
## 44  -0.72670483 0.528099906 -3.837728e-01 2.788895e-01 -2.026704e-01
## 45  -1.36828104 1.872193017 -2.561686e+00 3.505107e+00 -4.795971e+00
## 46   0.43281803 0.187331444  8.108043e-02 3.509307e-02  1.518891e-02
## 47  -0.81139318 0.658358886 -5.341879e-01 4.334364e-01 -3.516874e-01
## 48   1.44410126 2.085428454  3.011570e+00 4.349012e+00  6.280413e+00
## 49  -0.43144620 0.186145826 -8.031191e-02 3.465027e-02 -1.494973e-02
## 50   0.65564788 0.429874147  2.818461e-01 1.847918e-01  1.211583e-01
## 51   0.32192527 0.103635876  3.336301e-02 1.074039e-02  3.457604e-03
## 52  -0.78383894 0.614403485 -4.815934e-01 3.774916e-01 -2.958926e-01
## 53   1.57572752 2.482917217  3.912401e+00 6.164878e+00  9.714168e+00
## 54   0.64289931 0.413319517  2.657228e-01 1.708330e-01  1.098284e-01
## 55   0.08976065 0.008056974  7.231992e-04 6.491482e-05  5.826797e-06
## 56   0.27655075 0.076480316  2.115069e-02 5.849239e-03  1.617611e-03
## 57   0.67928882 0.461433296  3.134465e-01 2.129207e-01  1.446346e-01
## 58   0.08983289 0.008069948  7.249467e-04 6.512405e-05  5.850282e-06
## 59  -2.99309008 8.958588246 -2.681386e+01 8.025630e+01 -2.402143e+02
## 60   0.28488295 0.081158297  2.312062e-02 6.586669e-03  1.876430e-03
## 61  -0.36723464 0.134861283 -4.952574e-02 1.818757e-02 -6.679104e-03
## 62   0.18523056 0.034310362  6.355328e-03 1.177201e-03  2.180536e-04
## 63   0.58182373 0.338518850  1.969583e-01 1.145950e-01  6.667410e-02
## 64   1.39973683 1.959263186  2.742453e+00 3.838712e+00  5.373187e+00
## 65  -0.72729206 0.528953740 -3.847039e-01 2.797921e-01 -2.034905e-01
## 66   1.30254263 1.696617308  2.209916e+00 2.878510e+00  3.749382e+00
## 67   0.33584812 0.112793960  3.788164e-02 1.272248e-02  4.272820e-03
## 68   1.03850610 1.078494917  1.120024e+00 1.163151e+00  1.207940e+00
## 69   0.92072857 0.847741096  7.805394e-01 7.186650e-01  6.616954e-01
## 70   0.72087816 0.519665326  3.746154e-01 2.700521e-01  1.946746e-01
## 71  -1.04311894 1.088097120 -1.135015e+00 1.183955e+00 -1.235006e+00
## 72  -0.09018639 0.008133584 -7.335386e-04 6.615519e-05 -5.966298e-06
## 73   0.62351816 0.388774898  2.424082e-01 1.511459e-01  9.424223e-02
## 74  -0.95352336 0.909206794 -8.669499e-01 8.266570e-01 -7.882368e-01
## 75  -0.54282881 0.294663122 -1.599516e-01 8.682636e-02 -4.713185e-02
## 76   0.58099650 0.337556930  1.961194e-01 1.139447e-01  6.620146e-02
## 77   0.76817874 0.590098573  4.533012e-01 3.482163e-01  2.674924e-01
## 78   0.46376759 0.215080376  9.974731e-02 4.625957e-02  2.145369e-02
## 79  -0.88577630 0.784599649 -6.949798e-01 6.155966e-01 -5.452809e-01
## 80  -1.09978090 1.209518025 -1.330205e+00 1.462934e+00 -1.608907e+00
## 81   1.51270701 2.288282498  3.461501e+00 5.236237e+00  7.920892e+00
## 82   0.25792144 0.066523468  1.715783e-02 4.425372e-03  1.141398e-03
## 83   0.08844023 0.007821674  6.917507e-04 6.117859e-05  5.410648e-06
## 84  -0.12089654 0.014615973 -1.767021e-03 2.136267e-04 -2.582672e-05
## 85  -1.19432890 1.426421510 -1.703616e+00 2.034678e+00 -2.430075e+00
## 86   0.61199690 0.374540203  2.292174e-01 1.402804e-01  8.585115e-02
## 87  -0.21713985 0.047149713 -1.023808e-02 2.223095e-03 -4.827226e-04
## 88  -0.18275671 0.033400014 -6.104076e-03 1.115561e-03 -2.038762e-04
## 89   0.93334633 0.871135369  8.130710e-01 7.588768e-01  7.082949e-01
## 90   0.82177311 0.675311045  5.549525e-01 4.560450e-01  3.747655e-01
## 91   1.39211638 1.937988004  2.697905e+00 3.755798e+00  5.228507e+00
## 92  -0.47617392 0.226741605 -1.079684e-01 5.141176e-02 -2.448094e-02
## 93   0.65034856 0.422953250  2.750670e-01 1.788895e-01  1.163405e-01
## 94   1.39111046 1.935188302  2.692061e+00 3.744954e+00  5.209644e+00
## 95  -1.11078888 1.233851935 -1.370549e+00 1.522391e+00 -1.691055e+00
## 96  -0.86079259 0.740963878 -6.378162e-01 5.490275e-01 -4.725988e-01
## 97  -1.13173868 1.280832442 -1.449568e+00 1.640532e+00 -1.856653e+00
## 98  -1.45921400 2.129305496 -3.107112e+00 4.533942e+00 -6.615991e+00
## 99   0.07998255 0.006397209  5.116651e-04 4.092428e-05  3.273228e-06
## 100  0.65320434 0.426675909  2.787066e-01 1.820523e-01  1.189174e-01
##              X^6           X^7          X^8           X^9         X^10
## 1   6.639659e+00  9.102696e+00 1.247942e+01  1.710876e+01 2.345540e+01
## 2   3.242637e-02 -1.831111e-02 1.034025e-02 -5.839120e-03 3.297341e-03
## 3   2.292775e-03  8.325716e-04 3.023304e-04  1.097848e-04 3.986597e-05
## 4   6.424756e-02  4.065988e-02 2.573211e-02  1.628489e-02 1.030610e-02
## 5   4.365342e-03  1.764770e-03 7.134404e-04  2.884214e-04 1.165996e-04
## 6   1.428546e-06 -1.516038e-07 1.608888e-08 -1.707424e-09 1.811996e-10
## 7   1.192578e+01  1.802608e+01 2.724682e+01  4.118416e+01 6.225077e+01
## 8   7.194035e-07 -6.809804e-08 6.446095e-09 -6.101812e-10 5.775916e-11
## 9   6.761982e+01  1.364855e+02 2.754855e+02  5.560464e+02 1.122337e+03
## 10  6.084027e-08 -3.815543e-09 2.392883e-10 -1.500675e-11 9.411349e-13
## 11  4.936314e+00  6.441247e+00 8.404987e+00  1.096741e+01 1.431104e+01
## 12  1.429529e+02  3.268826e+02 7.474645e+02  1.709186e+03 3.908303e+03
## 13  7.177152e+00 -9.968064e+00 1.384425e+01 -1.922774e+01 2.670465e+01
## 14  4.695173e-04 -1.308962e-04 3.649238e-05 -1.017367e-05 2.836304e-06
## 15  5.615623e-06 -7.486824e-07 9.981533e-08 -1.330751e-08 1.774176e-09
## 16  6.615146e-02  4.206905e-02 2.675383e-02  1.701411e-02 1.082013e-02
## 17  5.275087e-04 -1.499459e-04 4.262256e-05 -1.211559e-05 3.443891e-06
## 18  3.514109e+02 -9.335073e+02 2.479821e+03 -6.587533e+03 1.749949e+04
## 19  2.112699e+02 -5.155971e+02 1.258298e+03 -3.070834e+03 7.494269e+03
## 20  5.292579e+00  6.986804e+00 9.223373e+00  1.217590e+01 1.607357e+01
## 21  8.313060e-04 -2.549105e-04 7.816539e-05 -2.396853e-05 7.349675e-06
## 22  3.194734e+01 -5.690807e+01 1.013708e+02 -1.805727e+02 3.216557e+02
## 23  2.581775e-05 -4.438519e-06 7.630584e-07 -1.311830e-07 2.255263e-08
## 24  3.211884e+00  3.901395e+00 4.738926e+00  5.756253e+00 6.991975e+00
## 25  4.633629e+01  8.781624e+01 1.664288e+02  3.154147e+02 5.977719e+02
## 26  6.362856e-03 -2.739013e-03 1.179061e-03 -5.075492e-04 2.184843e-04
## 27  2.899537e-04 -7.459620e-05 1.919132e-05 -4.937339e-06 1.270226e-06
## 28  3.004380e+01 -5.297212e+01 9.339849e+01 -1.646768e+02 2.903520e+02
## 29  9.486334e-03  4.364637e-03 2.008158e-03  9.239482e-04 4.251061e-04
## 30  6.871618e-02 -4.397800e-02 2.814569e-02 -1.801310e-02 1.152829e-02
## 31  8.925755e-03  4.065236e-03 1.851512e-03  8.432715e-04 3.840681e-04
## 32  1.226121e-01  8.642160e-02 6.091317e-02  4.293388e-02 3.026140e-02
## 33  1.229993e+00  1.273170e+00 1.317863e+00  1.364125e+00 1.412010e+00
## 34  5.097870e-02 -3.104227e-02 1.890246e-02 -1.151021e-02 7.008868e-03
## 35  1.657741e-02  8.370848e-03 4.226903e-03  2.134396e-03 1.077774e-03
## 36  2.562329e+01 -4.399542e+01 7.554051e+01 -1.297037e+02 2.227024e+02
## 37  2.330352e-01 -1.828066e-01 1.434043e-01 -1.124948e-01 8.824753e-02
## 38  3.795722e-01 -3.229809e-01 2.748269e-01 -2.338523e-01 1.989867e-01
## 39  1.979920e+02 -4.779939e+02 1.153977e+03 -2.785939e+03 6.725835e+03
## 40  2.221644e-09  8.025158e-11 2.898896e-12  1.047157e-13 3.782604e-15
## 41  7.641623e-05  1.574164e-05 3.242755e-06  6.680030e-07 1.376077e-07
## 42  2.215424e-03 -7.998948e-04 2.888079e-04 -1.042762e-04 3.764968e-05
## 43  1.899225e-01  1.439922e-01 1.091696e-01  8.276839e-02 6.275195e-02
## 44  1.472815e-01 -1.070302e-01 7.777936e-02 -5.652264e-02 4.107527e-02
## 45  6.562236e+00 -8.978983e+00 1.228577e+01 -1.681039e+01 2.300134e+01
## 46  6.574035e-03  2.845361e-03 1.231524e-03  5.330256e-04 2.307031e-04
## 47  2.853567e-01 -2.315365e-01 1.878671e-01 -1.524341e-01 1.236840e-01
## 48  9.069553e+00  1.309735e+01 1.891390e+01  2.731359e+01 3.944359e+01
## 49  6.450003e-03 -2.782829e-03 1.200641e-03 -5.180120e-04 2.234943e-04
## 50  7.943721e-02  5.208284e-02 3.414800e-02  2.238907e-02 1.467934e-02
## 51  1.113090e-03  3.583319e-04 1.153561e-04  3.713604e-05 1.195503e-05
## 52  2.319322e-01 -1.817975e-01 1.424999e-01 -1.116970e-01 8.755246e-02
## 53  1.530688e+01  2.411947e+01 3.800572e+01  5.988666e+01 9.436506e+01
## 54  7.060862e-02  4.539423e-02 2.918392e-02  1.876232e-02 1.206228e-02
## 55  5.230170e-07  4.694635e-08 4.213934e-09  3.782455e-10 3.395156e-11
## 56  4.473516e-04  1.237154e-04 3.421359e-05  9.461795e-06 2.616666e-06
## 57  9.824869e-02  6.673924e-02 4.533522e-02  3.079571e-02 2.091918e-02
## 58  5.255477e-07  4.721147e-08 4.241142e-09  3.809941e-10 3.422580e-11
## 59  7.189832e+02 -2.151981e+03 6.441074e+03 -1.927872e+04 5.770293e+04
## 60  5.345629e-04  1.522878e-04 4.338421e-05  1.235942e-05 3.520989e-06
## 61  2.452798e-03 -9.007526e-04 3.307875e-04 -1.214766e-04 4.461043e-05
## 62  4.039019e-05  7.481498e-06 1.385802e-06  2.566929e-07 4.754737e-08
## 63  3.879257e-02  2.257044e-02 1.313202e-02  7.640519e-03 4.445435e-03
## 64  7.521048e+00  1.052749e+01 1.473571e+01  2.062612e+01 2.887114e+01
## 65  1.479971e-01 -1.076371e-01 7.828360e-02 -5.693504e-02 4.140840e-02
## 66  4.883730e+00  6.361267e+00 8.285821e+00  1.079264e+01 1.405787e+01
## 67  1.435019e-03  4.819483e-04 1.618614e-04  5.436086e-05 1.825699e-05
## 68  1.254453e+00  1.302757e+00 1.352921e+00  1.405017e+00 1.459118e+00
## 69  6.092418e-01  5.609464e-01 5.164793e-01  4.755373e-01 4.378408e-01
## 70  1.403367e-01  1.011657e-01 7.292811e-02  5.257228e-02 3.789821e-02
## 71  1.288258e+00 -1.343807e+00 1.401750e+00 -1.462192e+00 1.525240e+00
## 72  5.380788e-07 -4.852739e-08 4.376510e-09 -3.947016e-10 3.559671e-11
## 73  5.876174e-02  3.663901e-02 2.284509e-02  1.424433e-02 8.881597e-03
## 74  7.516022e-01 -7.166702e-01 6.833618e-01 -6.516014e-01 6.213172e-01
## 75  2.558452e-02 -1.388802e-02 7.538816e-03 -4.092287e-03 2.221411e-03
## 76  3.846282e-02  2.234676e-02 1.298339e-02  7.543304e-03 4.382633e-03
## 77  2.054820e-01  1.578469e-01 1.212546e-01  9.314521e-02 7.155217e-02
## 78  9.949525e-03  4.614267e-03 2.139948e-03  9.924384e-04 4.602607e-04
## 79  4.829969e-01 -4.278272e-01 3.789592e-01 -3.356731e-01 2.973312e-01
## 80  1.769445e+00 -1.946002e+00 2.140175e+00 -2.353724e+00 2.588581e+00
## 81  1.198199e+01  1.812524e+01 2.741818e+01  4.147567e+01 6.274053e+01
## 82  2.943911e-04  7.592977e-05 1.958392e-05  5.051112e-06 1.302790e-06
## 83  4.785190e-07  4.232033e-08 3.742819e-09  3.310158e-10 2.927511e-11
## 84  3.122361e-06 -3.774827e-07 4.563635e-08 -5.517277e-09 6.670197e-10
## 85  2.902309e+00 -3.466311e+00 4.139916e+00 -4.944421e+00 5.905265e+00
## 86  5.254064e-02  3.215471e-02 1.967858e-02  1.204323e-02 7.370420e-03
## 87  1.048183e-04 -2.276023e-05 4.942153e-06 -1.073138e-06 2.330211e-07
## 88  3.725975e-05 -6.809469e-06 1.244476e-06 -2.274364e-07 4.156552e-08
## 89  6.610844e-01  6.170207e-01 5.758940e-01  5.375086e-01 5.016817e-01
## 90  3.079722e-01  2.530833e-01 2.079770e-01  1.709099e-01 1.404492e-01
## 91  7.278691e+00  1.013278e+01 1.410601e+01  1.963721e+01 2.733729e+01
## 92  1.165718e-02 -5.550847e-03 2.643169e-03 -1.258608e-03 5.993163e-04
## 93  7.566188e-02  4.920659e-02 3.200144e-02  2.081209e-02 1.353511e-02
## 94  7.247191e+00  1.008164e+01 1.402468e+01  1.950988e+01 2.714039e+01
## 95  1.878405e+00 -2.086511e+00 2.317673e+00 -2.574446e+00 2.859665e+00
## 96  4.068095e-01 -3.501786e-01 3.014312e-01 -2.594697e-01 2.233496e-01
## 97  2.101246e+00 -2.378062e+00 2.691344e+00 -3.045899e+00 3.447161e+00
## 98  9.654147e+00 -1.408747e+01 2.055663e+01 -2.999652e+01 4.377134e+01
## 99  2.618012e-07  2.093953e-08 1.674797e-09  1.339545e-10 1.071402e-11
## 100 7.767734e-02  5.073918e-02 3.314305e-02  2.164919e-02 1.414134e-02
##              Y_2
## 1    12.64964835
## 2     0.80835140
## 3     1.39778400
## 4     5.80157088
## 5     1.93896805
## 6     0.80846987
## 7    13.13508002
## 8     0.61074520
## 9    23.61466017
## 10    0.93863812
## 11   10.51666779
## 12   30.38179379
## 13   -2.47316994
## 14   -0.20680543
## 15   -1.02788366
## 16    3.59158255
## 17   -0.22677713
## 18   -8.89992060
## 19  -10.30688467
## 20   10.88355528
## 21   -1.25431877
## 22   -5.12043964
## 23    0.66298036
## 24    8.39042732
## 25   20.67435104
## 26   -0.42882644
## 27   -0.27013271
## 28   -5.57790175
## 29    1.67632108
## 30   -0.18341901
## 31    3.44331681
## 32    3.96538614
## 33    7.35730265
## 34    0.81190856
## 35    4.59343376
## 36   -4.41384777
## 37   -0.72268191
## 38    0.48076877
## 39   -9.12651981
## 40    1.05855516
## 41    1.62550099
## 42   -0.75719445
## 43    4.41522963
## 44   -0.53713230
## 45   -2.33601217
## 46    3.86758341
## 47   -1.13264251
## 48   12.08156152
## 49    0.69450371
## 50    3.05216961
## 51    2.16571208
## 52   -2.15584805
## 53   15.77258753
## 54    3.74741408
## 55    0.81827376
## 56    0.76551123
## 57    4.26641748
## 58    0.48608132
## 59  -17.40944773
## 60    3.32776132
## 61   -0.05703297
## 62    0.55888536
## 63    3.78267406
## 64   11.49745127
## 65    0.08134099
## 66   11.94320081
## 67    1.27832141
## 68    7.84718198
## 69    6.32310540
## 70    5.47214611
## 71   -1.31795543
## 72    1.58159354
## 73    2.14545663
## 74    0.78035252
## 75    0.66566615
## 76    3.46344676
## 77    3.48902741
## 78    3.56421953
## 79   -0.29991550
## 80   -1.21686709
## 81   13.72764289
## 82    1.33986011
## 83    1.65046252
## 84    0.95942965
## 85   -1.71301947
## 86    2.47805189
## 87    1.13339063
## 88    1.06662245
## 89    5.51907413
## 90    3.77630572
## 91   11.95518855
## 92   -0.42809498
## 93    4.32463092
## 94   10.44176619
## 95   -2.19438222
## 96    0.34750864
## 97   -0.87934387
## 98   -1.63965586
## 99    3.06848219
## 100   4.22049282
x_data_2
##               X         X.2           X.3          X.4           X.5
## 1    1.37095845 1.879527064  2.576754e+00 3.532622e+00  4.843078e+00
## 2   -0.56469817 0.318884025 -1.800732e-01 1.016870e-01 -5.742247e-02
## 3    0.36312841 0.131862243  4.788293e-02 1.738765e-02  6.313950e-03
## 4    0.63286260 0.400515077  2.534710e-01 1.604123e-01  1.015190e-01
## 5    0.40426832 0.163432877  6.607074e-02 2.671031e-02  1.079813e-02
## 6   -0.10612452 0.011262413 -1.195218e-03 1.268419e-04 -1.346104e-05
## 7    1.51152200 2.284698749  3.453372e+00 5.219848e+00  7.889916e+00
## 8   -0.09465904 0.008960334 -8.481766e-04 8.028758e-05 -7.599945e-06
## 9    2.01842371 4.074034289  8.223127e+00 1.659776e+01  3.350130e+01
## 10  -0.06271410 0.003933058 -2.466582e-04 1.546895e-05 -9.701211e-07
## 11   1.30486965 1.702684815  2.221782e+00 2.899136e+00  3.782994e+00
## 12   2.28664539 5.228747152  1.195629e+01 2.733980e+01  6.251642e+01
## 13  -1.38886070 1.928934047 -2.679021e+00 3.720787e+00 -5.167654e+00
## 14  -0.27878877 0.077723177 -2.166835e-02 6.040892e-03 -1.684133e-03
## 15  -0.13332134 0.017774579 -2.369731e-03 3.159356e-04 -4.212096e-05
## 16   0.63595040 0.404432909  2.571993e-01 1.635660e-01  1.040198e-01
## 17  -0.28425292 0.080799723 -2.296756e-02 6.528595e-03 -1.855772e-03
## 18  -2.65645542 7.056755403 -1.874596e+01 4.979780e+01 -1.322856e+02
## 19  -2.44046693 5.955878829 -1.453513e+01 3.547249e+01 -8.656945e+01
## 20   1.32011335 1.742699246  2.300561e+00 3.037001e+00  4.009185e+00
## 21  -0.30663859 0.094027227 -2.883238e-02 8.841119e-03 -2.711028e-03
## 22  -1.78130843 3.173059737 -5.652198e+00 1.006831e+01 -1.793476e+01
## 23  -0.17191736 0.029555577 -5.081117e-03 8.735321e-04 -1.501753e-04
## 24   1.21467470 1.475434625  1.792173e+00 2.176907e+00  2.644234e+00
## 25   1.89519346 3.591758256  6.807077e+00 1.290073e+01  2.444937e+01
## 26  -0.43046913 0.185303673 -7.976751e-02 3.433745e-02 -1.478121e-02
## 27  -0.25726938 0.066187535 -1.702803e-02 4.380790e-03 -1.127043e-03
## 28  -1.76316309 3.108744065 -5.481223e+00 9.664290e+00 -1.703972e+01
## 29   0.46009735 0.211689576  9.739781e-02 4.481248e-02  2.061810e-02
## 30  -0.63999488 0.409593441 -2.621377e-01 1.677668e-01 -1.073699e-01
## 31   0.45545012 0.207434815  9.447621e-02 4.302920e-02  1.959766e-02
## 32   0.70483734 0.496795672  3.501601e-01 2.468059e-01  1.739580e-01
## 33   1.03510352 1.071439301  1.109051e+00 1.147982e+00  1.188280e+00
## 34  -0.60892638 0.370791331 -2.257846e-01 1.374862e-01 -8.371898e-02
## 35   0.50495512 0.254979677  1.287533e-01 6.501464e-02  3.282947e-02
## 36  -1.71700868 2.948118804 -5.061946e+00 8.691404e+00 -1.492322e+01
## 37  -0.78445901 0.615375936 -4.827372e-01 3.786875e-01 -2.970649e-01
## 38  -0.85090759 0.724043734 -6.160943e-01 5.242393e-01 -4.460792e-01
## 39  -2.41420765 5.828398577 -1.407096e+01 3.397023e+01 -8.201119e+01
## 40   0.03612261 0.001304843  4.713432e-05 1.702615e-06  6.150288e-08
## 41   0.20599860 0.042435423  8.741638e-03 1.800765e-03  3.709551e-04
## 42  -0.36105730 0.130362373 -4.706829e-02 1.699435e-02 -6.135933e-03
## 43   0.75816324 0.574811492  4.358009e-01 3.304083e-01  2.505034e-01
## 44  -0.72670483 0.528099906 -3.837728e-01 2.788895e-01 -2.026704e-01
## 45  -1.36828104 1.872193017 -2.561686e+00 3.505107e+00 -4.795971e+00
## 46   0.43281803 0.187331444  8.108043e-02 3.509307e-02  1.518891e-02
## 47  -0.81139318 0.658358886 -5.341879e-01 4.334364e-01 -3.516874e-01
## 48   1.44410126 2.085428454  3.011570e+00 4.349012e+00  6.280413e+00
## 49  -0.43144620 0.186145826 -8.031191e-02 3.465027e-02 -1.494973e-02
## 50   0.65564788 0.429874147  2.818461e-01 1.847918e-01  1.211583e-01
## 51   0.32192527 0.103635876  3.336301e-02 1.074039e-02  3.457604e-03
## 52  -0.78383894 0.614403485 -4.815934e-01 3.774916e-01 -2.958926e-01
## 53   1.57572752 2.482917217  3.912401e+00 6.164878e+00  9.714168e+00
## 54   0.64289931 0.413319517  2.657228e-01 1.708330e-01  1.098284e-01
## 55   0.08976065 0.008056974  7.231992e-04 6.491482e-05  5.826797e-06
## 56   0.27655075 0.076480316  2.115069e-02 5.849239e-03  1.617611e-03
## 57   0.67928882 0.461433296  3.134465e-01 2.129207e-01  1.446346e-01
## 58   0.08983289 0.008069948  7.249467e-04 6.512405e-05  5.850282e-06
## 59  -2.99309008 8.958588246 -2.681386e+01 8.025630e+01 -2.402143e+02
## 60   0.28488295 0.081158297  2.312062e-02 6.586669e-03  1.876430e-03
## 61  -0.36723464 0.134861283 -4.952574e-02 1.818757e-02 -6.679104e-03
## 62   0.18523056 0.034310362  6.355328e-03 1.177201e-03  2.180536e-04
## 63   0.58182373 0.338518850  1.969583e-01 1.145950e-01  6.667410e-02
## 64   1.39973683 1.959263186  2.742453e+00 3.838712e+00  5.373187e+00
## 65  -0.72729206 0.528953740 -3.847039e-01 2.797921e-01 -2.034905e-01
## 66   1.30254263 1.696617308  2.209916e+00 2.878510e+00  3.749382e+00
## 67   0.33584812 0.112793960  3.788164e-02 1.272248e-02  4.272820e-03
## 68   1.03850610 1.078494917  1.120024e+00 1.163151e+00  1.207940e+00
## 69   0.92072857 0.847741096  7.805394e-01 7.186650e-01  6.616954e-01
## 70   0.72087816 0.519665326  3.746154e-01 2.700521e-01  1.946746e-01
## 71  -1.04311894 1.088097120 -1.135015e+00 1.183955e+00 -1.235006e+00
## 72  -0.09018639 0.008133584 -7.335386e-04 6.615519e-05 -5.966298e-06
## 73   0.62351816 0.388774898  2.424082e-01 1.511459e-01  9.424223e-02
## 74  -0.95352336 0.909206794 -8.669499e-01 8.266570e-01 -7.882368e-01
## 75  -0.54282881 0.294663122 -1.599516e-01 8.682636e-02 -4.713185e-02
## 76   0.58099650 0.337556930  1.961194e-01 1.139447e-01  6.620146e-02
## 77   0.76817874 0.590098573  4.533012e-01 3.482163e-01  2.674924e-01
## 78   0.46376759 0.215080376  9.974731e-02 4.625957e-02  2.145369e-02
## 79  -0.88577630 0.784599649 -6.949798e-01 6.155966e-01 -5.452809e-01
## 80  -1.09978090 1.209518025 -1.330205e+00 1.462934e+00 -1.608907e+00
## 81   1.51270701 2.288282498  3.461501e+00 5.236237e+00  7.920892e+00
## 82   0.25792144 0.066523468  1.715783e-02 4.425372e-03  1.141398e-03
## 83   0.08844023 0.007821674  6.917507e-04 6.117859e-05  5.410648e-06
## 84  -0.12089654 0.014615973 -1.767021e-03 2.136267e-04 -2.582672e-05
## 85  -1.19432890 1.426421510 -1.703616e+00 2.034678e+00 -2.430075e+00
## 86   0.61199690 0.374540203  2.292174e-01 1.402804e-01  8.585115e-02
## 87  -0.21713985 0.047149713 -1.023808e-02 2.223095e-03 -4.827226e-04
## 88  -0.18275671 0.033400014 -6.104076e-03 1.115561e-03 -2.038762e-04
## 89   0.93334633 0.871135369  8.130710e-01 7.588768e-01  7.082949e-01
## 90   0.82177311 0.675311045  5.549525e-01 4.560450e-01  3.747655e-01
## 91   1.39211638 1.937988004  2.697905e+00 3.755798e+00  5.228507e+00
## 92  -0.47617392 0.226741605 -1.079684e-01 5.141176e-02 -2.448094e-02
## 93   0.65034856 0.422953250  2.750670e-01 1.788895e-01  1.163405e-01
## 94   1.39111046 1.935188302  2.692061e+00 3.744954e+00  5.209644e+00
## 95  -1.11078888 1.233851935 -1.370549e+00 1.522391e+00 -1.691055e+00
## 96  -0.86079259 0.740963878 -6.378162e-01 5.490275e-01 -4.725988e-01
## 97  -1.13173868 1.280832442 -1.449568e+00 1.640532e+00 -1.856653e+00
## 98  -1.45921400 2.129305496 -3.107112e+00 4.533942e+00 -6.615991e+00
## 99   0.07998255 0.006397209  5.116651e-04 4.092428e-05  3.273228e-06
## 100  0.65320434 0.426675909  2.787066e-01 1.820523e-01  1.189174e-01
##              X.6           X.7          X.8           X.9         X.10
## 1   6.639659e+00  9.102696e+00 1.247942e+01  1.710876e+01 2.345540e+01
## 2   3.242637e-02 -1.831111e-02 1.034025e-02 -5.839120e-03 3.297341e-03
## 3   2.292775e-03  8.325716e-04 3.023304e-04  1.097848e-04 3.986597e-05
## 4   6.424756e-02  4.065988e-02 2.573211e-02  1.628489e-02 1.030610e-02
## 5   4.365342e-03  1.764770e-03 7.134404e-04  2.884214e-04 1.165996e-04
## 6   1.428546e-06 -1.516038e-07 1.608888e-08 -1.707424e-09 1.811996e-10
## 7   1.192578e+01  1.802608e+01 2.724682e+01  4.118416e+01 6.225077e+01
## 8   7.194035e-07 -6.809804e-08 6.446095e-09 -6.101812e-10 5.775916e-11
## 9   6.761982e+01  1.364855e+02 2.754855e+02  5.560464e+02 1.122337e+03
## 10  6.084027e-08 -3.815543e-09 2.392883e-10 -1.500675e-11 9.411349e-13
## 11  4.936314e+00  6.441247e+00 8.404987e+00  1.096741e+01 1.431104e+01
## 12  1.429529e+02  3.268826e+02 7.474645e+02  1.709186e+03 3.908303e+03
## 13  7.177152e+00 -9.968064e+00 1.384425e+01 -1.922774e+01 2.670465e+01
## 14  4.695173e-04 -1.308962e-04 3.649238e-05 -1.017367e-05 2.836304e-06
## 15  5.615623e-06 -7.486824e-07 9.981533e-08 -1.330751e-08 1.774176e-09
## 16  6.615146e-02  4.206905e-02 2.675383e-02  1.701411e-02 1.082013e-02
## 17  5.275087e-04 -1.499459e-04 4.262256e-05 -1.211559e-05 3.443891e-06
## 18  3.514109e+02 -9.335073e+02 2.479821e+03 -6.587533e+03 1.749949e+04
## 19  2.112699e+02 -5.155971e+02 1.258298e+03 -3.070834e+03 7.494269e+03
## 20  5.292579e+00  6.986804e+00 9.223373e+00  1.217590e+01 1.607357e+01
## 21  8.313060e-04 -2.549105e-04 7.816539e-05 -2.396853e-05 7.349675e-06
## 22  3.194734e+01 -5.690807e+01 1.013708e+02 -1.805727e+02 3.216557e+02
## 23  2.581775e-05 -4.438519e-06 7.630584e-07 -1.311830e-07 2.255263e-08
## 24  3.211884e+00  3.901395e+00 4.738926e+00  5.756253e+00 6.991975e+00
## 25  4.633629e+01  8.781624e+01 1.664288e+02  3.154147e+02 5.977719e+02
## 26  6.362856e-03 -2.739013e-03 1.179061e-03 -5.075492e-04 2.184843e-04
## 27  2.899537e-04 -7.459620e-05 1.919132e-05 -4.937339e-06 1.270226e-06
## 28  3.004380e+01 -5.297212e+01 9.339849e+01 -1.646768e+02 2.903520e+02
## 29  9.486334e-03  4.364637e-03 2.008158e-03  9.239482e-04 4.251061e-04
## 30  6.871618e-02 -4.397800e-02 2.814569e-02 -1.801310e-02 1.152829e-02
## 31  8.925755e-03  4.065236e-03 1.851512e-03  8.432715e-04 3.840681e-04
## 32  1.226121e-01  8.642160e-02 6.091317e-02  4.293388e-02 3.026140e-02
## 33  1.229993e+00  1.273170e+00 1.317863e+00  1.364125e+00 1.412010e+00
## 34  5.097870e-02 -3.104227e-02 1.890246e-02 -1.151021e-02 7.008868e-03
## 35  1.657741e-02  8.370848e-03 4.226903e-03  2.134396e-03 1.077774e-03
## 36  2.562329e+01 -4.399542e+01 7.554051e+01 -1.297037e+02 2.227024e+02
## 37  2.330352e-01 -1.828066e-01 1.434043e-01 -1.124948e-01 8.824753e-02
## 38  3.795722e-01 -3.229809e-01 2.748269e-01 -2.338523e-01 1.989867e-01
## 39  1.979920e+02 -4.779939e+02 1.153977e+03 -2.785939e+03 6.725835e+03
## 40  2.221644e-09  8.025158e-11 2.898896e-12  1.047157e-13 3.782604e-15
## 41  7.641623e-05  1.574164e-05 3.242755e-06  6.680030e-07 1.376077e-07
## 42  2.215424e-03 -7.998948e-04 2.888079e-04 -1.042762e-04 3.764968e-05
## 43  1.899225e-01  1.439922e-01 1.091696e-01  8.276839e-02 6.275195e-02
## 44  1.472815e-01 -1.070302e-01 7.777936e-02 -5.652264e-02 4.107527e-02
## 45  6.562236e+00 -8.978983e+00 1.228577e+01 -1.681039e+01 2.300134e+01
## 46  6.574035e-03  2.845361e-03 1.231524e-03  5.330256e-04 2.307031e-04
## 47  2.853567e-01 -2.315365e-01 1.878671e-01 -1.524341e-01 1.236840e-01
## 48  9.069553e+00  1.309735e+01 1.891390e+01  2.731359e+01 3.944359e+01
## 49  6.450003e-03 -2.782829e-03 1.200641e-03 -5.180120e-04 2.234943e-04
## 50  7.943721e-02  5.208284e-02 3.414800e-02  2.238907e-02 1.467934e-02
## 51  1.113090e-03  3.583319e-04 1.153561e-04  3.713604e-05 1.195503e-05
## 52  2.319322e-01 -1.817975e-01 1.424999e-01 -1.116970e-01 8.755246e-02
## 53  1.530688e+01  2.411947e+01 3.800572e+01  5.988666e+01 9.436506e+01
## 54  7.060862e-02  4.539423e-02 2.918392e-02  1.876232e-02 1.206228e-02
## 55  5.230170e-07  4.694635e-08 4.213934e-09  3.782455e-10 3.395156e-11
## 56  4.473516e-04  1.237154e-04 3.421359e-05  9.461795e-06 2.616666e-06
## 57  9.824869e-02  6.673924e-02 4.533522e-02  3.079571e-02 2.091918e-02
## 58  5.255477e-07  4.721147e-08 4.241142e-09  3.809941e-10 3.422580e-11
## 59  7.189832e+02 -2.151981e+03 6.441074e+03 -1.927872e+04 5.770293e+04
## 60  5.345629e-04  1.522878e-04 4.338421e-05  1.235942e-05 3.520989e-06
## 61  2.452798e-03 -9.007526e-04 3.307875e-04 -1.214766e-04 4.461043e-05
## 62  4.039019e-05  7.481498e-06 1.385802e-06  2.566929e-07 4.754737e-08
## 63  3.879257e-02  2.257044e-02 1.313202e-02  7.640519e-03 4.445435e-03
## 64  7.521048e+00  1.052749e+01 1.473571e+01  2.062612e+01 2.887114e+01
## 65  1.479971e-01 -1.076371e-01 7.828360e-02 -5.693504e-02 4.140840e-02
## 66  4.883730e+00  6.361267e+00 8.285821e+00  1.079264e+01 1.405787e+01
## 67  1.435019e-03  4.819483e-04 1.618614e-04  5.436086e-05 1.825699e-05
## 68  1.254453e+00  1.302757e+00 1.352921e+00  1.405017e+00 1.459118e+00
## 69  6.092418e-01  5.609464e-01 5.164793e-01  4.755373e-01 4.378408e-01
## 70  1.403367e-01  1.011657e-01 7.292811e-02  5.257228e-02 3.789821e-02
## 71  1.288258e+00 -1.343807e+00 1.401750e+00 -1.462192e+00 1.525240e+00
## 72  5.380788e-07 -4.852739e-08 4.376510e-09 -3.947016e-10 3.559671e-11
## 73  5.876174e-02  3.663901e-02 2.284509e-02  1.424433e-02 8.881597e-03
## 74  7.516022e-01 -7.166702e-01 6.833618e-01 -6.516014e-01 6.213172e-01
## 75  2.558452e-02 -1.388802e-02 7.538816e-03 -4.092287e-03 2.221411e-03
## 76  3.846282e-02  2.234676e-02 1.298339e-02  7.543304e-03 4.382633e-03
## 77  2.054820e-01  1.578469e-01 1.212546e-01  9.314521e-02 7.155217e-02
## 78  9.949525e-03  4.614267e-03 2.139948e-03  9.924384e-04 4.602607e-04
## 79  4.829969e-01 -4.278272e-01 3.789592e-01 -3.356731e-01 2.973312e-01
## 80  1.769445e+00 -1.946002e+00 2.140175e+00 -2.353724e+00 2.588581e+00
## 81  1.198199e+01  1.812524e+01 2.741818e+01  4.147567e+01 6.274053e+01
## 82  2.943911e-04  7.592977e-05 1.958392e-05  5.051112e-06 1.302790e-06
## 83  4.785190e-07  4.232033e-08 3.742819e-09  3.310158e-10 2.927511e-11
## 84  3.122361e-06 -3.774827e-07 4.563635e-08 -5.517277e-09 6.670197e-10
## 85  2.902309e+00 -3.466311e+00 4.139916e+00 -4.944421e+00 5.905265e+00
## 86  5.254064e-02  3.215471e-02 1.967858e-02  1.204323e-02 7.370420e-03
## 87  1.048183e-04 -2.276023e-05 4.942153e-06 -1.073138e-06 2.330211e-07
## 88  3.725975e-05 -6.809469e-06 1.244476e-06 -2.274364e-07 4.156552e-08
## 89  6.610844e-01  6.170207e-01 5.758940e-01  5.375086e-01 5.016817e-01
## 90  3.079722e-01  2.530833e-01 2.079770e-01  1.709099e-01 1.404492e-01
## 91  7.278691e+00  1.013278e+01 1.410601e+01  1.963721e+01 2.733729e+01
## 92  1.165718e-02 -5.550847e-03 2.643169e-03 -1.258608e-03 5.993163e-04
## 93  7.566188e-02  4.920659e-02 3.200144e-02  2.081209e-02 1.353511e-02
## 94  7.247191e+00  1.008164e+01 1.402468e+01  1.950988e+01 2.714039e+01
## 95  1.878405e+00 -2.086511e+00 2.317673e+00 -2.574446e+00 2.859665e+00
## 96  4.068095e-01 -3.501786e-01 3.014312e-01 -2.594697e-01 2.233496e-01
## 97  2.101246e+00 -2.378062e+00 2.691344e+00 -3.045899e+00 3.447161e+00
## 98  9.654147e+00 -1.408747e+01 2.055663e+01 -2.999652e+01 4.377134e+01
## 99  2.618012e-07  2.093953e-08 1.674797e-09  1.339545e-10 1.071402e-11
## 100 7.767734e-02  5.073918e-02 3.314305e-02  2.164919e-02 1.414134e-02
##               Y_2
## 1    2.495771e+01
## 2    1.998973e+00
## 3   -1.127218e-03
## 4    2.950132e+00
## 5    3.376385e-01
## 6    1.105513e+00
## 7    4.564295e+01
## 8    8.776497e-01
## 9    3.424018e+02
## 10   1.119161e+00
## 11   1.707802e+01
## 12   8.183145e+02
## 13  -2.440560e+01
## 14   4.954556e-01
## 15  -6.611010e-01
## 16   7.228389e-01
## 17   4.869749e-01
## 18  -2.330066e+03
## 19  -1.289355e+03
## 20   1.860427e+01
## 21  -4.942623e-01
## 22  -1.427406e+02
## 23   1.124691e+00
## 24   9.756848e+00
## 25   2.205388e+02
## 26   5.648936e-01
## 27   3.861419e-01
## 28  -1.334550e+02
## 29  -2.138364e-01
## 30   1.069571e+00
## 31   1.577784e+00
## 32   7.231767e-01
## 33   4.182989e+00
## 34   2.045284e+00
## 35   2.460783e+00
## 36  -1.100857e+02
## 37   4.256640e-01
## 38   1.394046e+00
## 39  -1.194454e+03
## 40   9.475305e-01
## 41   9.139321e-01
## 42   1.103212e-01
## 43   9.152966e-01
## 44   7.029796e-01
## 45  -2.186133e+01
## 46   2.120499e+00
## 47  -5.983408e-02
## 48   3.331021e+01
## 49   1.689906e+00
## 50   7.383868e-02
## 51   9.601974e-01
## 52  -1.006039e+00
## 53   6.246586e+01
## 54   8.398399e-01
## 55   5.321548e-01
## 56  -2.379430e-01
## 57   1.159086e+00
## 58   1.997179e-01
## 59  -5.379487e+03
## 60   2.288056e+00
## 61   8.222222e-01
## 62  -7.176368e-02
## 63   1.219633e+00
## 64   2.695598e+01
## 65   1.320921e+00
## 66   1.833559e+01
## 67   8.512360e-03
## 68   4.711542e+00
## 69   2.487264e+00
## 70   2.148480e+00
## 71  -2.589295e+00
## 72   1.836619e+00
## 73  -6.534583e-01
## 74   8.977834e-01
## 75   1.830058e+00
## 76   9.050909e-01
## 77  -5.438995e-02
## 78   1.654544e+00
## 79   4.136259e-01
## 80  -3.871360e+00
## 81   4.646455e+01
## 82   4.160809e-01
## 83   1.368807e+00
## 84   1.294653e+00
## 85  -7.945038e+00
## 86  -2.558499e-01
## 87   1.700692e+00
## 88   1.554180e+00
## 89   1.706245e+00
## 90   3.812008e-02
## 91   2.653692e+01
## 92   6.410349e-01
## 93   1.375628e+00
## 94   2.491010e+01
## 95  -5.175448e+00
## 96   1.210328e+00
## 97  -4.541379e+00
## 98  -3.363218e+01
## 99   2.815228e+00
## 100  1.255669e+00

Subset Selection)

#Exhaustive
b<-regsubsets(Y_2~., data = x_data_2, nvmax = 10)
mysummary(b)
## Subset selection object
## Call: regsubsets.formula(Y_2 ~ ., data = x_data_2, nvmax = 10)
## 10 Variables  (and intercept)
##      Forced in Forced out
## X        FALSE      FALSE
## X.2      FALSE      FALSE
## X.3      FALSE      FALSE
## X.4      FALSE      FALSE
## X.5      FALSE      FALSE
## X.6      FALSE      FALSE
## X.7      FALSE      FALSE
## X.8      FALSE      FALSE
## X.9      FALSE      FALSE
## X.10     FALSE      FALSE
## 1 subsets of each size up to 10
## Selection Algorithm: exhaustive
##           X   X.2 X.3 X.4 X.5 X.6 X.7 X.8 X.9 X.10
## 1  ( 1 )  " " " " " " " " " " " " "*" " " " " " " 
## 2  ( 1 )  " " " " "*" " " " " " " "*" " " " " " " 
## 3  ( 1 )  " " " " " " " " "*" " " "*" " " "*" " " 
## 4  ( 1 )  " " " " "*" " " "*" " " "*" " " "*" " " 
## 5  ( 1 )  " " " " "*" "*" "*" " " "*" " " "*" " " 
## 6  ( 1 )  " " " " "*" " " "*" " " "*" "*" "*" "*" 
## 7  ( 1 )  "*" " " " " "*" "*" "*" "*" "*" " " "*" 
## 8  ( 1 )  " " " " "*" "*" "*" "*" "*" "*" "*" "*" 
## 9  ( 1 )  " " "*" "*" "*" "*" "*" "*" "*" "*" "*" 
## 10  ( 1 ) "*" "*" "*" "*" "*" "*" "*" "*" "*" "*" 
## [1] "Cps: "
##  [1] 17.503967 18.089401 12.245486  6.777278  4.776474  4.643057  5.317350
##  [8]  7.095679  9.031722 11.000000

## [1] "Optimal model variable count:"
## [1] 6

## [1] "BICs: "
##  [1] -1295.638 -1292.287 -1294.937 -1297.765 -1297.382 -1295.103 -1291.971
##  [8] -1287.615 -1283.081 -1278.512

## [1] "Optimal model variable count:"
## [1] 4
## [1] "R-squared: "
##  [1] 0.9999978 0.9999979 0.9999980 0.9999982 0.9999982 0.9999983 0.9999983
##  [8] 0.9999983 0.9999983 0.9999983

## [1] "Optimal model variable count:"
## [1] 10

Performing exhaustive subset selection via regsubsets, we see that the optimal model according to Cps is the 6 cofactor model with X to the power of 3, 5, 7, 8, 9, and 10. For tihs model, the Cp is at a minimum of 4.643057.

According to the BIC measure the optimal model is the 4 cofactor model with X to the power of 3, 5, 7, and 9. For this model, the BIC is at a minimum of -1297.765.

Similar to the previous problem, the optimal model according to the R-squared is unsuprisingly the model with the most amount of cofactors. Here, all models considered have a very large r-squared, since the one variable model sufficiently explains the response variable Y (since it was made with only one variable X^7).

f)

#fit lasso reg. model
x_dataM_2 = as.matrix(x_data_2)

lasso.mod_2=glmnet(x_dataM_2[,-11],x_dataM_2[ ,11],alpha=1, lambda = grid)
plot(lasso.mod_2, label=TRUE)

#Cross-validation
set.seed(42)
#x_dataf_2 = as.data.frame(x_data_2) 
cv.out_2=cv.glmnet(x_dataM_2[,-11],x_dataM_2[ ,11],alpha=1)
plot(cv.out_2)

#find lambda corresponding to smallest mse
bestlam_2 =cv.out_2$lambda.min
bestlam_2
## [1] 19.60987
#extract coeficients for model with this lambda
coef(cv.out_2, s = "lambda.min")
## 11 x 1 sparse Matrix of class "dgCMatrix"
##                     1
## (Intercept) -1.958884
## X            .       
## X.2          .       
## X.3          .       
## X.4          .       
## X.5          .       
## X.6          .       
## X.7          2.419947
## X.8          .       
## X.9          .       
## X.10         .

The lambda that gives the smallest cross-validated mean-squared error is 19.60987. This lambda corresponds with the point on the cross-validation graph whose log(Lambda) = log(19.60987) = 1.2924, which is the point between the two calculated vertical dotted lines. The coefficient values can be found in the 11 x 1 spase matrix above. Notice that X^7 is the only covariate with non-zero coefficients, and it coefficient is 2.5. Remember our data was created using the real coefficient valeu of 2.5. Thus, this model is accurate in its assumptions, though it underestimates the X^2 coefficient. Again, the fact that this particular covariate has a non-zero coeficient in the optimal model is not suprising, since our data was generated using exclusively this X^7 variable multiplied by 2.5.

a)

indexes = sample(1:nrow(uni), size=0.5*nrow(uni))
test = uni[indexes,]
train = uni[-indexes,]

b)

uniTrainingLM <- lm(Apps ~ ., train)
summary(uniTrainingLM)
## 
## Call:
## lm(formula = Apps ~ ., data = train)
## 
## Residuals:
##     Min      1Q  Median      3Q     Max 
## -3249.2  -497.2   -82.7   368.1  6922.8 
## 
## Coefficients:
##               Estimate Std. Error t value Pr(>|t|)    
## (Intercept) -2.712e+02  6.109e+02  -0.444 0.657341    
## PrivateYes  -7.848e+02  2.131e+02  -3.682 0.000265 ***
## Accept       1.263e+00  7.158e-02  17.646  < 2e-16 ***
## Enroll       7.895e-02  2.844e-01   0.278 0.781475    
## Top10perc    5.412e+01  7.620e+00   7.102 6.32e-12 ***
## Top25perc   -1.387e+01  6.431e+00  -2.158 0.031602 *  
## F.Undergrad  2.522e-03  4.889e-02   0.052 0.958883    
## P.Undergrad  4.884e-03  5.655e-02   0.086 0.931222    
## Outstate    -3.523e-02  2.886e-02  -1.221 0.222874    
## Room.Board   2.259e-01  7.154e-02   3.157 0.001723 ** 
## Books       -2.342e-02  3.523e-01  -0.066 0.947022    
## Personal    -7.133e-03  9.378e-02  -0.076 0.939410    
## PhD         -5.904e+00  7.265e+00  -0.813 0.416918    
## Terminal    -6.962e+00  8.137e+00  -0.856 0.392780    
## S.F.Ratio    1.459e+00  1.840e+01   0.079 0.936845    
## perc.alumni -1.103e+01  6.294e+00  -1.753 0.080418 .  
## Expend       4.928e-02  1.626e-02   3.031 0.002611 ** 
## Grad.Rate    9.541e+00  4.388e+00   2.175 0.030295 *  
## ---
## Signif. codes:  0 '***' 0.001 '**' 0.01 '*' 0.05 '.' 0.1 ' ' 1
## 
## Residual standard error: 1098 on 371 degrees of freedom
## Multiple R-squared:  0.9216, Adjusted R-squared:  0.918 
## F-statistic: 256.6 on 17 and 371 DF,  p-value: < 2.2e-16
#Calculate MSE on training set
mean(residuals(uniTrainingLM)^2)
## [1] 1150171
#Calculate MSE on test set by calculating residuals
mean(((test$Apps)-predict(uniTrainingLM,test))^2)
## [1] 1209637

The mean square error of the linear model is 1.209636610^{6}

c)

#Convert data to format usable by glmnet package, split into appropriate test and training data sets
x=model.matrix(Apps~.,uni)[,-1]
xTest = x[indexes,]
xTrain = x[-indexes,]

y=uni$Apps
yTest = uni[indexes,]$Apps
yTrain = uni[-indexes,]$Apps
ridge.cv = cv.glmnet(xTrain,yTrain,alpha=0)
lambda.cv = ridge.cv$lambda.min
lambda.cv
## [1] 392.5473
fit.ridge = glmnet(xTrain,yTrain,alpha=0,lambda=lambda.cv)
pred.ridge = predict(fit.ridge,newx=xTest)
mean((yTest-pred.ridge)^2)
## [1] 2039168

The lambda minimized by cross-validation in the ridge regression model is 392.5473434, and the resulting test error is 2.03916810^{6}

d)

lasso.cv = cv.glmnet(xTrain,yTrain,alpha=1)
lambda.cv = lasso.cv$lambda.min
lambda.cv
## [1] 11.17989
fit.lasso = glmnet(xTrain,yTrain,alpha=1,lambda=lambda.cv)
pred.lasso = predict(fit.lasso,newx=xTest)
mean((yTest-pred.lasso)^2)
## [1] 1227317

The lambda minimized by cross-validation in the LASSO model is 11.1798891, and the resulting test error is 1.227316710^{6}

e)

fit.pcr = pcr(Apps~.,data=train,scale=TRUE,validation="CV")
summary(fit.pcr)
## Data:    X dimension: 389 17 
##  Y dimension: 389 1
## Fit method: svdpc
## Number of components considered: 17
## 
## VALIDATION: RMSEP
## Cross-validated using 10 random segments.
##        (Intercept)  1 comps  2 comps  3 comps  4 comps  5 comps  6 comps
## CV            3841     3872     1900     1897     1628     1480     1451
## adjCV         3841     3878     1896     1895     1501     1474     1447
##        7 comps  8 comps  9 comps  10 comps  11 comps  12 comps  13 comps
## CV        1453     1449     1350      1339      1342      1342      1343
## adjCV     1450     1459     1336      1334      1337      1338      1339
##        14 comps  15 comps  16 comps  17 comps
## CV         1336      1342      1184      1185
## adjCV      1331      1338      1178      1179
## 
## TRAINING: % variance explained
##       1 comps  2 comps  3 comps  4 comps  5 comps  6 comps  7 comps
## X      31.371    57.65    65.46    71.25    76.87    81.56    85.11
## Apps    1.626    76.68    76.87    86.56    86.72    87.01    87.01
##       8 comps  9 comps  10 comps  11 comps  12 comps  13 comps  14 comps
## X       88.03    90.81     93.33     95.41     97.19     98.17     98.90
## Apps    87.21    89.06     89.20     89.20     89.23     89.24     89.37
##       15 comps  16 comps  17 comps
## X        99.48     99.86    100.00
## Apps     89.40     92.10     92.16
pcr_pred <- predict(fit.pcr, test)
mean((pcr_pred - test$Apps)^2)
## [1] 3623155
validationplot(fit.pcr,val.type="MSEP")

Appears the smallest Cross Validation error occurs when all components are included in the model (M = 17)

f)

mean(((test$Apps)-predict(uniTrainingLM,test))^2)  # Least Squares MSE
## [1] 1209637
mean((yTest-pred.ridge)^2)  # Ridge MSE
## [1] 2039168
mean((yTest-pred.lasso)^2)  # LASSO MSE
## [1] 1227317
mean((pcr_pred - test$Apps)^2)  # PCR MSE
## [1] 3623155
coef(uniTrainingLM)
##   (Intercept)    PrivateYes        Accept        Enroll     Top10perc 
## -2.712196e+02 -7.848468e+02  1.263133e+00  7.895531e-02  5.411756e+01 
##     Top25perc   F.Undergrad   P.Undergrad      Outstate    Room.Board 
## -1.387474e+01  2.522176e-03  4.884264e-03 -3.523260e-02  2.258838e-01 
##         Books      Personal           PhD      Terminal     S.F.Ratio 
## -2.342423e-02 -7.132978e-03 -5.903813e+00 -6.961877e+00  1.458547e+00 
##   perc.alumni        Expend     Grad.Rate 
## -1.103375e+01  4.928336e-02  9.540876e+00
coef(fit.ridge)
## 18 x 1 sparse Matrix of class "dgCMatrix"
##                        s0
## (Intercept) -1.310206e+03
## PrivateYes  -6.941615e+02
## Accept       7.606190e-01
## Enroll       7.650760e-01
## Top10perc    2.987822e+01
## Top25perc    2.016432e+00
## F.Undergrad  9.180116e-02
## P.Undergrad -1.187016e-03
## Outstate     1.111003e-02
## Room.Board   2.669129e-01
## Books        2.647735e-02
## Personal    -6.344657e-02
## PhD         -9.017944e-01
## Terminal    -8.043214e+00
## S.F.Ratio    4.032825e+00
## perc.alumni -1.640348e+01
## Expend       5.410772e-02
## Grad.Rate    1.052146e+01
coef(fit.lasso)
## 18 x 1 sparse Matrix of class "dgCMatrix"
##                        s0
## (Intercept) -3.849310e+02
## PrivateYes  -7.627745e+02
## Accept       1.251083e+00
## Enroll       9.984464e-02
## Top10perc    4.803665e+01
## Top25perc   -8.839646e+00
## F.Undergrad  2.877663e-03
## P.Undergrad  .           
## Outstate    -2.213359e-02
## Room.Board   2.001993e-01
## Books        .           
## Personal     .           
## PhD         -4.473273e+00
## Terminal    -6.889357e+00
## S.F.Ratio    .           
## perc.alumni -1.076730e+01
## Expend       4.575306e-02
## Grad.Rate    7.870912e+00
coef(fit.pcr)
## , , 17 comps
## 
##                    Apps
## PrivateYes  -351.932393
## Accept      3164.692435
## Enroll        78.990563
## Top10perc   1001.943589
## Top25perc   -282.169693
## F.Undergrad   13.136146
## P.Undergrad    6.921946
## Outstate    -145.545916
## Room.Board   259.997785
## Books         -3.995510
## Personal      -4.882939
## PhD          -98.764668
## Terminal    -102.093346
## S.F.Ratio      5.981821
## perc.alumni -138.326075
## Expend       277.576700
## Grad.Rate    164.551336

We can measure the accuracy of a model through the Mean Squared Error calculated using a separate test set. This measures the average squared residual and can assess how well a model fits a data set, in these cases test sets. The Least Squares Regression and the LASSO model both tend to have similar test errors, which are the lowest. The ridge regression model tends to have higher test error, with the PCR model having the worst test error on average. If these procedures were repeated for a different split between training and test sets, the trends usually hold but the values themselves change.

attach(Boston)

head(Boston)
##      crim zn indus chas   nox    rm  age    dis rad tax ptratio  black
## 1 0.00632 18  2.31    0 0.538 6.575 65.2 4.0900   1 296    15.3 396.90
## 2 0.02731  0  7.07    0 0.469 6.421 78.9 4.9671   2 242    17.8 396.90
## 3 0.02729  0  7.07    0 0.469 7.185 61.1 4.9671   2 242    17.8 392.83
## 4 0.03237  0  2.18    0 0.458 6.998 45.8 6.0622   3 222    18.7 394.63
## 5 0.06905  0  2.18    0 0.458 7.147 54.2 6.0622   3 222    18.7 396.90
## 6 0.02985  0  2.18    0 0.458 6.430 58.7 6.0622   3 222    18.7 394.12
##   lstat medv
## 1  4.98 24.0
## 2  9.14 21.6
## 3  4.03 34.7
## 4  2.94 33.4
## 5  5.33 36.2
## 6  5.21 28.7
#Initial Summary
summary(Boston)
##       crim                zn             indus            chas        
##  Min.   : 0.00632   Min.   :  0.00   Min.   : 0.46   Min.   :0.00000  
##  1st Qu.: 0.08204   1st Qu.:  0.00   1st Qu.: 5.19   1st Qu.:0.00000  
##  Median : 0.25651   Median :  0.00   Median : 9.69   Median :0.00000  
##  Mean   : 3.61352   Mean   : 11.36   Mean   :11.14   Mean   :0.06917  
##  3rd Qu.: 3.67708   3rd Qu.: 12.50   3rd Qu.:18.10   3rd Qu.:0.00000  
##  Max.   :88.97620   Max.   :100.00   Max.   :27.74   Max.   :1.00000  
##       nox               rm             age              dis        
##  Min.   :0.3850   Min.   :3.561   Min.   :  2.90   Min.   : 1.130  
##  1st Qu.:0.4490   1st Qu.:5.886   1st Qu.: 45.02   1st Qu.: 2.100  
##  Median :0.5380   Median :6.208   Median : 77.50   Median : 3.207  
##  Mean   :0.5547   Mean   :6.285   Mean   : 68.57   Mean   : 3.795  
##  3rd Qu.:0.6240   3rd Qu.:6.623   3rd Qu.: 94.08   3rd Qu.: 5.188  
##  Max.   :0.8710   Max.   :8.780   Max.   :100.00   Max.   :12.127  
##       rad              tax           ptratio          black       
##  Min.   : 1.000   Min.   :187.0   Min.   :12.60   Min.   :  0.32  
##  1st Qu.: 4.000   1st Qu.:279.0   1st Qu.:17.40   1st Qu.:375.38  
##  Median : 5.000   Median :330.0   Median :19.05   Median :391.44  
##  Mean   : 9.549   Mean   :408.2   Mean   :18.46   Mean   :356.67  
##  3rd Qu.:24.000   3rd Qu.:666.0   3rd Qu.:20.20   3rd Qu.:396.23  
##  Max.   :24.000   Max.   :711.0   Max.   :22.00   Max.   :396.90  
##      lstat            medv      
##  Min.   : 1.73   Min.   : 5.00  
##  1st Qu.: 6.95   1st Qu.:17.02  
##  Median :11.36   Median :21.20  
##  Mean   :12.65   Mean   :22.53  
##  3rd Qu.:16.95   3rd Qu.:25.00  
##  Max.   :37.97   Max.   :50.00
#remove factor like variables
Boston_cont = subset(Boston, select=-c(rad, chas))

#Pairplot
pairs(Boston_cont, cex = .5)

#Boxplot for factor var. rad
boxplot(crim~rad, main="Crime Rate by Rad", 
xlab="Rad", ylab="Crime Rate")

#Boxplot for factor var. chas
boxplot(crim~chas, main="Crime Rate by Chas", 
xlab="Chas", ylab="Crime Rate")

Boston_to_trans = subset(Boston, select=c(crim,age, dis, medv))

#before transform
pairs(Boston_to_trans)

#transform
Boston_to_trans[,2:4] = log(Boston_to_trans[,2:4])

#after transform
pairs(Boston_to_trans)

#zn indus tax and ptratio all seem to only have high crime rates at one value. See which value that is 
boxplot(crim~zn, main="Crime Rate by zn", xlab="zn", ylab="Crime Rate")

boxplot(crim~indus, main="Crime Rate by indus", xlab="indus", ylab="Crime Rate")

boxplot(crim~tax, main="Crime Rate by tax", xlab="tax", ylab="Crime Rate")

boxplot(crim~ptratio, main="Crime Rate by ptratio", xlab="ptratio", ylab="Crime Rate")

#zn = 0 , indus = 18.10 , 

Looking at the pairs plot, zn, indus, age, dis, tax, and ptratio seem to be the strongest possible contenders for covariates to be included in our model. This is of course based on their correlation with the crim variable (the top row). For the two factor covariates rad and chas, rad value of 24 seems to be correlated with larger crime rate and while the mean crim rate of chas=0 points is low, any point with a large crime rate had a chas=0. In orther words, very little of the chas=1 points had a large crime rate.

#fit lasso reg. model
Boston_M = as.matrix(Boston)
lambdas=seq(1e-3,1e3,length=100)
lasso.mod_3=glmnet(Boston_M[,-1],Boston_M[ ,1],alpha=1, lambda = lambdas)
plot(lasso.mod_3, label=TRUE)

#Cross-validation
set.seed(42)
#Boston_df = as.data.frame(Boston_df) 
cv.out_3=cv.glmnet(Boston_M[,-1],Boston_M[ ,1],alpha=1)
plot(cv.out_3)

#find lambda corresponding to smallest mse
bestlam =cv.out_3$lambda.min
bestlam
## [1] 0.05630926
#extract coeficients for model with this lambda
coef(cv.out_3, s = "lambda.min")
## 14 x 1 sparse Matrix of class "dgCMatrix"
##                        1
## (Intercept) 12.319178096
## zn           0.035726832
## indus       -0.068876055
## chas        -0.577832639
## nox         -6.631559478
## rm           0.208676938
## age          .          
## dis         -0.768388825
## rad          0.512333871
## tax          .          
## ptratio     -0.179631375
## black       -0.007551172
## lstat        0.124630014
## medv        -0.154550130
x = Boston_M[,-1]
y=Boston_M[ ,1]
finalModel1=glmnet(x,y,alpha=1, lambda = 0.05630926)
#plotres(finalModel1, which=1)


pred.lasso = predict(finalModel1,newx=x)
mean((y-pred.lasso)^2)
## [1] 40.48466
summary(finalModel1)
##           Length Class     Mode   
## a0         1     -none-    numeric
## beta      13     dgCMatrix S4     
## df         1     -none-    numeric
## dim        2     -none-    numeric
## lambda     1     -none-    numeric
## dev.ratio  1     -none-    numeric
## nulldev    1     -none-    numeric
## npasses    1     -none-    numeric
## jerr       1     -none-    numeric
## offset     1     -none-    logical
## call       5     -none-    call   
## nobs       1     -none-    numeric
fits <- fitted(finalModel1)
## calculate the deviance residuals
resids  <- (pred.lasso-y)
fits
## NULL
plot(pred.lasso, resids, main = "Model Resids")
abline(h=0, lty=2)

#QQ to see if residuals follow normal
qqnorm(y-pred.lasso)
qqline(y-pred.lasso)

We use cross validation via the ‘cv.glmnet’ funciton in order to validate our model selection and paremeter (lambda) setting. The minimizing lambda for this lasso regression is 0.0563092. This lambda corresponds with the point on the cross-validation graph whose log(Lambda) = log(0.0563092.) = -1.24942, which is the point between the two calculated vertical dotted lines. The covariates determined not useful and thus whose coefficients are set to zero in the model are age and tax. While these two variables looked as though they may have been correlated with crime rate, they could have been left out of the model due to multicollinearity. We validated our model using cross validation and considered a large amount of lambdas as values. The optimal lambda fell comfortably in between the max and min of all the lambas considered.

The residuals are not optmial, but they do have decent amounts both above and below the y=0 line. They also, for the majority of the middle instances, follow the qq line in the last plot. They only stray from it in the early and the late theoretical quantiles.